2023-07-13T01:48:04.892 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-13T01:48:04.893 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_gabrioux@teuthology --verbose --description orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} --name gabrioux-2023-07-13_00:38:31-orch:cephadm-cv-refactor-osd-objectstore-distro-default-smithi --archive /home/teuthworker/archive/gabrioux-2023-07-13_00:38:31-orch:cephadm-cv-refactor-osd-objectstore-distro-default-smithi/7335617 -- /home/teuthworker/archive/gabrioux-2023-07-13_00:38:31-orch:cephadm-cv-refactor-osd-objectstore-distro-default-smithi/7335617/orig.config.yaml 2023-07-13T01:48:04.922 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-13T01:48:05.007 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2023-07-13_00:38:31-orch:cephadm-cv-refactor-osd-objectstore-distro-default-smithi/7335617 branch: cv-refactor-osd-objectstore description: orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} email: gabrioux@redhat.com first_in_suite: false job_id: '7335617' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2023-07-13_00:38:31-orch:cephadm-cv-refactor-osd-objectstore-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: cv-refactor-osd-objectstore ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 3990f91ac125621431ef1b77c2fc96421bec9e51 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 3990f91ac125621431ef1b77c2fc96421bec9e51 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: cv-refactor-osd-objectstore sha1: 3990f91ac125621431ef1b77c2fc96421bec9e51 owner: scheduled_gabrioux@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a sha1: 3990f91ac125621431ef1b77c2fc96421bec9e51 sleep_before_teardown: 0 suite: orch:cephadm suite_branch: cv-refactor-osd-objectstore suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3990f91ac125621431ef1b77c2fc96421bec9e51/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3990f91ac125621431ef1b77c2fc96421bec9e51 targets: smithi032.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkuioc2rB2/ZTZ+AFnxgD9c4H7/EH7hXBXfONd5YUN24Waof4JzkkRP1BqAM1FozKfo/ofJ+PzGw1gJmk9zmuqkLw3IOO/FiA0PL+GnZT778QaDf1dZghcPsnB5FfyWV1zutiz/anIzgnVhwMjUyjvPcdMxpM+7sj/1WH5w+GDWze5LUtrxdz8WsE8ksBqjzRojkDaekwrIiF5td3u6P0KAXXNJEDJA079oG4/RkIqvdQ528YZvsIaplCdOrxdW4cjLrw05CgRhaO1RsGF2CFj82/fokfTYoy411zJAvQi68IIWCSLSXXPKUHV+gPZzBsi1b8SyWVmntQdswCiGpcGTr+HgxRaYS/fXBGucFx3M9VOUB7aPUg9QF5ZdgBIRvdDCIiY3g3B9o8y/iLRNbfEqw1p3LYe3iSYXI0j3iUA4hH3Veaisaiv5BWwe1aP6dgv0X0e+JG5GyItClhDX5b/SXVZvlFX/s4QNpaoh1Kj+tkQ6Tk5Fc0+j2Hihm4hhWc= smithi191.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHUDwEPbZnAEsrJfqXeQeXDFMxSMHER0Ti13d2UX1k+VHmo+1j2Adq//dylnODETskRCo01l8iJe1AZkohtm5uQOJNgLPgUdczL6qor26lqvvHVbffHAgqG8BpA732n7iRPqxjL1hmgAS/334HbwaDYHUdy+x59oI30zLCClOj+fYyIaEdX47LqCkO/TFZMIkZUeIVqPBSJjNnjgPtNp5I4qulMhsk9hjC4/rBnkvpFN8XMLdge4lqPwzc4aH4XEEjvLswlawEdUIVEZlnmH0Hzq7S9QliGBbQH2RFtHa2T+le+u6zhF/eZOe/TPwjBr5ZnvTSRRs7XlNAMRESBFJg9nkWcxHG4JPSyzP6tYgrxKhyq9TpMyh7LyzLLv2hBA/3Mljd85p5g9cI1GSZus1f2v/KY0hdt8HVghubPtdsQXpJod+DjBS38IUcePNKAK8MyRqayYJX2pu3zvXPW4SPhvW8bK9EDbRT3cDvg5ilnxAjegu6fppdXH9PpG/mwtc= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - sudo sed -i 's/runtime = "runc"/#runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/#runtime = "crun"/runtime = "crun"/g' /usr/share/containers/containers.conf - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b53f236c1077f1d295baafb9e0ca9a4b447019d7 timestamp: 2023-07-13_00:38:31 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-13T01:48:05.008 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3990f91ac125621431ef1b77c2fc96421bec9e51/qa; will attempt to use it 2023-07-13T01:48:05.009 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3990f91ac125621431ef1b77c2fc96421bec9e51/qa/tasks 2023-07-13T01:48:05.009 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-13T01:48:05.011 INFO:teuthology.task.internal:Checking packages... 2023-07-13T01:48:05.041 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3990f91ac125621431ef1b77c2fc96421bec9e51' 2023-07-13T01:48:05.042 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-13T01:48:05.042 INFO:teuthology.packaging:ref: None 2023-07-13T01:48:05.042 INFO:teuthology.packaging:tag: None 2023-07-13T01:48:05.043 INFO:teuthology.packaging:branch: cv-refactor-osd-objectstore 2023-07-13T01:48:05.043 INFO:teuthology.packaging:sha1: 3990f91ac125621431ef1b77c2fc96421bec9e51 2023-07-13T01:48:05.044 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=cv-refactor-osd-objectstore 2023-07-13T01:48:05.632 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-4093.g3990f91a 2023-07-13T01:48:05.633 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-13T01:48:05.645 INFO:teuthology.task.internal:no buildpackages task found 2023-07-13T01:48:05.646 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-13T01:48:05.651 INFO:teuthology.task.internal:Saving configuration 2023-07-13T01:48:05.662 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-13T01:48:05.670 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-13T01:48:05.700 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2023-07-13_00:38:31-orch:cephadm-cv-refactor-osd-objectstore-distro-default-smithi/7335617', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-13 01:42:57.447577', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJjZc5LojQKd9h3H4fe2kud23/5TVCXjUyLon6k0QbwLl6M3iWMO1854lcXZXqHzSWwOf1LD4polHQKa4djj6KA='} 2023-07-13T01:48:05.724 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2023-07-13_00:38:31-orch:cephadm-cv-refactor-osd-objectstore-distro-default-smithi/7335617', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-13 01:42:57.446116', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDQqPOtrpLB+NfDDyp+8IpA45iuVKbhRsahDU+JT32E5ibMKI+5WEiRkGa/a8wCHxr1mQzPR8qTOwS6r15CVA2o='} 2023-07-13T01:48:05.724 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-13T01:48:05.730 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2023-07-13T01:48:05.731 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2023-07-13T01:48:05.731 INFO:teuthology.run_tasks:Running task console_log... 2023-07-13T01:48:05.810 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc14eff63a0>, signals=[15]) 2023-07-13T01:48:05.812 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-13T01:48:05.818 INFO:teuthology.task.internal:Opening connections... 2023-07-13T01:48:05.819 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi032.front.sepia.ceph.com 2023-07-13T01:48:05.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-13T01:48:05.891 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2023-07-13T01:48:05.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-13T01:48:05.958 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-13T01:48:05.965 DEBUG:teuthology.orchestra.run.smithi032:> uname -m 2023-07-13T01:48:05.989 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2023-07-13T01:48:05.989 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release 2023-07-13T01:48:06.050 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="CentOS Stream" 2023-07-13T01:48:06.051 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="8" 2023-07-13T01:48:06.051 INFO:teuthology.orchestra.run.smithi032.stdout:ID="centos" 2023-07-13T01:48:06.051 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="rhel fedora" 2023-07-13T01:48:06.051 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="8" 2023-07-13T01:48:06.051 INFO:teuthology.orchestra.run.smithi032.stdout:PLATFORM_ID="platform:el8" 2023-07-13T01:48:06.052 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="CentOS Stream 8" 2023-07-13T01:48:06.052 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31" 2023-07-13T01:48:06.052 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-07-13T01:48:06.052 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://centos.org/" 2023-07-13T01:48:06.052 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-13T01:48:06.052 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-13T01:48:06.053 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-07-13T01:48:06.053 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2023-07-13T01:48:06.086 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2023-07-13T01:48:06.108 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2023-07-13T01:48:06.109 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2023-07-13T01:48:06.173 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="CentOS Stream" 2023-07-13T01:48:06.174 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="8" 2023-07-13T01:48:06.174 INFO:teuthology.orchestra.run.smithi191.stdout:ID="centos" 2023-07-13T01:48:06.175 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="rhel fedora" 2023-07-13T01:48:06.175 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="8" 2023-07-13T01:48:06.175 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el8" 2023-07-13T01:48:06.176 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="CentOS Stream 8" 2023-07-13T01:48:06.176 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31" 2023-07-13T01:48:06.176 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-07-13T01:48:06.177 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://centos.org/" 2023-07-13T01:48:06.177 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-13T01:48:06.177 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-13T01:48:06.178 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-07-13T01:48:06.178 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2023-07-13T01:48:06.202 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-13T01:48:06.221 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-13T01:48:06.227 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-13T01:48:06.228 DEBUG:teuthology.orchestra.run.smithi032:> test '!' -e /home/ubuntu/cephtest 2023-07-13T01:48:06.231 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2023-07-13T01:48:06.252 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-07-13T01:48:06.259 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-07-13T01:48:06.259 DEBUG:teuthology.orchestra.run.smithi032:> test -z $(ls -A /var/lib/ceph) 2023-07-13T01:48:06.292 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2023-07-13T01:48:06.318 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-07-13T01:48:06.391 INFO:teuthology.run_tasks:Running task kernel... 2023-07-13T01:48:06.410 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-07-13T01:48:06.411 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-07-13T01:48:06.411 DEBUG:teuthology.orchestra.run.smithi032:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-13T01:48:06.412 DEBUG:teuthology.orchestra.run.smithi191:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-13T01:48:06.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-13T01:48:06.433 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2023-07-13T01:48:06.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-13T01:48:06.434 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2023-07-13T01:48:06.496 INFO:teuthology.orchestra.run.smithi191.stdout:4.18.0-489.el8.x86_64 2023-07-13T01:48:06.496 INFO:teuthology.task.kernel:Running kernel on smithi191: 4.18.0-489.el8.x86_64 2023-07-13T01:48:06.496 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum install -y kernel 2023-07-13T01:48:06.497 INFO:teuthology.orchestra.run.smithi032.stdout:4.18.0-489.el8.x86_64 2023-07-13T01:48:06.498 INFO:teuthology.task.kernel:Running kernel on smithi032: 4.18.0-489.el8.x86_64 2023-07-13T01:48:06.498 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum install -y kernel 2023-07-13T01:49:17.139 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - AppStream 462 kB/s | 31 MB 01:07 2023-07-13T01:49:26.430 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - BaseOS 12 MB/s | 41 MB 00:03 2023-07-13T01:49:33.563 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - Extras 23 kB/s | 18 kB 00:00 2023-07-13T01:49:34.155 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 8 - Extras common packages 16 kB/s | 6.6 kB 00:00 2023-07-13T01:49:36.873 INFO:teuthology.orchestra.run.smithi032.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-07-13T01:50:00.010 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 695 kB/s | 16 MB 00:22 2023-07-13T01:50:04.030 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 404 kB/s | 24 kB 00:00 2023-07-13T01:50:07.446 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-07-13T01:50:07.446 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-07-13T01:50:07.664 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2023-07-13T01:50:07.665 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2023-07-13T01:50:07.665 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2023-07-13T01:50:07.665 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2023-07-13T01:50:07.665 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2023-07-13T01:50:07.665 INFO:teuthology.orchestra.run.smithi032.stdout: kernel x86_64 4.18.0-500.el8 baseos 9.8 M 2023-07-13T01:50:07.665 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2023-07-13T01:50:07.666 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-core x86_64 4.18.0-500.el8 baseos 43 M 2023-07-13T01:50:07.666 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules x86_64 4.18.0-500.el8 baseos 35 M 2023-07-13T01:50:07.666 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T01:50:07.666 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2023-07-13T01:50:07.666 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2023-07-13T01:50:07.666 INFO:teuthology.orchestra.run.smithi032.stdout:Install 3 Packages 2023-07-13T01:50:07.666 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T01:50:07.667 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 88 M 2023-07-13T01:50:07.667 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 95 M 2023-07-13T01:50:07.667 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2023-07-13T01:50:09.715 INFO:teuthology.orchestra.run.smithi032.stdout:(1/3): kernel-4.18.0-500.el8.x86_64.rpm 7.1 MB/s | 9.8 MB 00:01 2023-07-13T01:50:10.840 INFO:teuthology.orchestra.run.smithi032.stdout:(2/3): kernel-modules-4.18.0-500.el8.x86_64.rpm 14 MB/s | 35 MB 00:02 2023-07-13T01:50:11.440 INFO:teuthology.orchestra.run.smithi032.stdout:(3/3): kernel-core-4.18.0-500.el8.x86_64.rpm 14 MB/s | 43 MB 00:03 2023-07-13T01:50:11.441 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2023-07-13T01:50:11.441 INFO:teuthology.orchestra.run.smithi032.stdout:Total 23 MB/s | 88 MB 00:03 2023-07-13T01:50:12.134 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2023-07-13T01:50:12.610 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2023-07-13T01:50:12.611 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2023-07-13T01:50:13.883 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2023-07-13T01:50:13.906 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2023-07-13T01:50:16.818 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2023-07-13T01:50:17.301 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-core-4.18.0-500.el8.x86_64 1/3 2023-07-13T01:50:21.075 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-core-4.18.0-500.el8.x86_64 1/3 2023-07-13T01:50:21.458 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-modules-4.18.0-500.el8.x86_64 2/3 2023-07-13T01:50:25.068 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-4.18.0-500.el8.x86_64 2/3 2023-07-13T01:50:25.407 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-4.18.0-500.el8.x86_64 3/3 2023-07-13T01:50:42.892 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 8 - AppStream 204 kB/s | 31 MB 02:33 2023-07-13T01:50:48.497 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-core-4.18.0-500.el8.x86_64 3/3 2023-07-13T01:50:48.521 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-4.18.0-500.el8.x86_64 3/3 2023-07-13T01:50:48.884 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-4.18.0-500.el8.x86_64 3/3 2023-07-13T01:50:48.885 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-4.18.0-500.el8.x86_64 1/3 2023-07-13T01:50:48.885 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-core-4.18.0-500.el8.x86_64 2/3 2023-07-13T01:50:50.028 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-modules-4.18.0-500.el8.x86_64 3/3 2023-07-13T01:50:50.029 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T01:50:50.029 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2023-07-13T01:50:50.029 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-4.18.0-500.el8.x86_64 kernel-core-4.18.0-500.el8.x86_64 2023-07-13T01:50:50.029 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules-4.18.0-500.el8.x86_64 2023-07-13T01:50:50.029 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T01:50:50.029 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2023-07-13T01:50:50.170 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel | sort -rV | head -n 1 2023-07-13T01:50:50.391 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-4.18.0-500.el8.x86_64 2023-07-13T01:50:50.391 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-500.el8.x86_64 2023-07-13T01:50:50.391 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-500.el8.x86_64 2023-07-13T01:50:50.392 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-07-13T01:50:50.392 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-07-13T01:50:50.392 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-07-13T01:50:50.392 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi032.front.sepia.ceph.com, path=None, version=distro) 2023-07-13T01:50:50.393 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel | sort -rV | head -n 1 2023-07-13T01:50:50.400 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 8 - BaseOS 23 MB/s | 41 MB 00:01 2023-07-13T01:50:50.608 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-4.18.0-500.el8.x86_64 2023-07-13T01:50:50.609 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-500.el8.x86_64 2023-07-13T01:50:50.609 DEBUG:teuthology.orchestra.run.smithi032:> sudo rpm -qi grub2-tools 2023-07-13T01:50:50.643 INFO:teuthology.orchestra.run.smithi032.stdout:Name : grub2-tools 2023-07-13T01:50:50.644 INFO:teuthology.orchestra.run.smithi032.stdout:Epoch : 1 2023-07-13T01:50:50.644 INFO:teuthology.orchestra.run.smithi032.stdout:Version : 2.02 2023-07-13T01:50:50.644 INFO:teuthology.orchestra.run.smithi032.stdout:Release : 129.el8 2023-07-13T01:50:50.644 INFO:teuthology.orchestra.run.smithi032.stdout:Architecture: x86_64 2023-07-13T01:50:50.644 INFO:teuthology.orchestra.run.smithi032.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-07-13T01:50:50.644 INFO:teuthology.orchestra.run.smithi032.stdout:Group : System Environment/Base 2023-07-13T01:50:50.645 INFO:teuthology.orchestra.run.smithi032.stdout:Size : 9265586 2023-07-13T01:50:50.645 INFO:teuthology.orchestra.run.smithi032.stdout:License : GPLv3+ 2023-07-13T01:50:50.645 INFO:teuthology.orchestra.run.smithi032.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-07-13T01:50:50.645 INFO:teuthology.orchestra.run.smithi032.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-07-13T01:50:50.645 INFO:teuthology.orchestra.run.smithi032.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-07-13T01:50:50.645 INFO:teuthology.orchestra.run.smithi032.stdout:Build Host : kbuilder.bsys.centos.org 2023-07-13T01:50:50.646 INFO:teuthology.orchestra.run.smithi032.stdout:Relocations : (not relocatable) 2023-07-13T01:50:50.646 INFO:teuthology.orchestra.run.smithi032.stdout:Packager : CentOS BuildSystem 2023-07-13T01:50:50.646 INFO:teuthology.orchestra.run.smithi032.stdout:Vendor : CentOS 2023-07-13T01:50:50.646 INFO:teuthology.orchestra.run.smithi032.stdout:URL : http://www.gnu.org/software/grub/ 2023-07-13T01:50:50.646 INFO:teuthology.orchestra.run.smithi032.stdout:Summary : Support tools for GRUB. 2023-07-13T01:50:50.646 INFO:teuthology.orchestra.run.smithi032.stdout:Description : 2023-07-13T01:50:50.647 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T01:50:50.647 INFO:teuthology.orchestra.run.smithi032.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-07-13T01:50:50.647 INFO:teuthology.orchestra.run.smithi032.stdout:customizable bootloader with modular architecture. It supports a rich 2023-07-13T01:50:50.647 INFO:teuthology.orchestra.run.smithi032.stdout:variety of kernel formats, file systems, computer architectures and 2023-07-13T01:50:50.647 INFO:teuthology.orchestra.run.smithi032.stdout:hardware devices. 2023-07-13T01:50:50.647 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T01:50:50.647 INFO:teuthology.orchestra.run.smithi032.stdout:This subpackage provides tools for support of all platforms. 2023-07-13T01:50:50.648 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-07-13T01:50:50.649 INFO:teuthology.task.kernel:Updating grub on smithi032 to boot 4.18.0-500.el8.x86_64 2023-07-13T01:50:50.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-07-13T01:50:52.068 INFO:teuthology.orchestra.run.smithi032.stderr:Generating grub configuration file ... 2023-07-13T01:50:53.922 INFO:teuthology.orchestra.run.smithi032.stderr:done 2023-07-13T01:50:53.924 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2023-07-13T01:50:53.964 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.2H8suTUHM3 2023-07-13T01:50:53.965 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2H8suTUHM3 2023-07-13T01:50:54.037 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.2H8suTUHM3 2023-07-13T01:50:54.179 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.2H8suTUHM3 is 6KB 2023-07-13T01:50:54.229 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.2H8suTUHM3 2023-07-13T01:50:54.247 DEBUG:teuthology.orchestra.run.smithi032:> sudo /bin/ls /boot/loader/entries || true 2023-07-13T01:50:54.321 INFO:teuthology.orchestra.run.smithi032.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-07-13T01:50:54.321 INFO:teuthology.orchestra.run.smithi032.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-07-13T01:50:54.322 INFO:teuthology.orchestra.run.smithi032.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-07-13T01:50:54.322 INFO:teuthology.orchestra.run.smithi032.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-500.el8.x86_64.conf 2023-07-13T01:50:54.323 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-500.el8.x86_64 2023-07-13T01:50:54.511 DEBUG:teuthology.orchestra.run.smithi032:> sudo shutdown -r now 2023-07-13T01:50:57.241 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 8 - Extras 23 kB/s | 18 kB 00:00 2023-07-13T01:50:57.709 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 6.6 kB 00:00 2023-07-13T01:50:59.969 INFO:teuthology.orchestra.run.smithi191.stdout:Copr repo for python3-asyncssh owned by ceph 1.6 kB/s | 3.5 kB 00:02 2023-07-13T01:51:01.305 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2023-07-13T01:51:05.095 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 458 kB/s | 24 kB 00:00 2023-07-13T01:51:08.227 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-07-13T01:51:08.228 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-07-13T01:51:08.432 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-07-13T01:51:08.432 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-07-13T01:51:08.433 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2023-07-13T01:51:08.433 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-07-13T01:51:08.433 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2023-07-13T01:51:08.433 INFO:teuthology.orchestra.run.smithi191.stdout: kernel x86_64 4.18.0-500.el8 baseos 9.8 M 2023-07-13T01:51:08.433 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2023-07-13T01:51:08.433 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core x86_64 4.18.0-500.el8 baseos 43 M 2023-07-13T01:51:08.433 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules x86_64 4.18.0-500.el8 baseos 35 M 2023-07-13T01:51:08.434 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-13T01:51:08.434 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-07-13T01:51:08.434 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-07-13T01:51:08.434 INFO:teuthology.orchestra.run.smithi191.stdout:Install 3 Packages 2023-07-13T01:51:08.434 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-13T01:51:08.435 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 88 M 2023-07-13T01:51:08.435 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 95 M 2023-07-13T01:51:08.435 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-07-13T01:51:12.563 INFO:teuthology.orchestra.run.smithi191.stdout:(1/3): kernel-4.18.0-500.el8.x86_64.rpm 2.5 MB/s | 9.8 MB 00:03 2023-07-13T01:51:17.121 INFO:teuthology.orchestra.run.smithi191.stdout:(2/3): kernel-modules-4.18.0-500.el8.x86_64.rpm 4.2 MB/s | 35 MB 00:08 2023-07-13T01:51:18.457 INFO:teuthology.orchestra.run.smithi191.stdout:(3/3): kernel-core-4.18.0-500.el8.x86_64.rpm 4.3 MB/s | 43 MB 00:09 2023-07-13T01:51:18.458 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-07-13T01:51:18.458 INFO:teuthology.orchestra.run.smithi191.stdout:Total 8.7 MB/s | 88 MB 00:10 2023-07-13T01:51:19.226 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-07-13T01:51:19.656 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-07-13T01:51:19.656 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-07-13T01:51:20.865 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-07-13T01:51:20.882 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-07-13T01:51:23.704 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-07-13T01:51:24.240 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-core-4.18.0-500.el8.x86_64 1/3 2023-07-13T01:51:24.515 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-07-13T01:51:24.515 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2023-07-13T01:51:24.516 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-13T01:51:27.873 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-4.18.0-500.el8.x86_64 1/3 2023-07-13T01:51:28.255 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-4.18.0-500.el8.x86_64 2/3 2023-07-13T01:51:31.619 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-4.18.0-500.el8.x86_64 2/3 2023-07-13T01:51:31.933 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-4.18.0-500.el8.x86_64 3/3 2023-07-13T01:51:42.964 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.32 2023-07-13T01:51:51.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2023-07-13T01:51:51.967 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-13T01:51:53.628 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-4.18.0-500.el8.x86_64 3/3 2023-07-13T01:51:53.651 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-4.18.0-500.el8.x86_64 3/3 2023-07-13T01:51:54.013 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-4.18.0-500.el8.x86_64 3/3 2023-07-13T01:51:54.014 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-4.18.0-500.el8.x86_64 1/3 2023-07-13T01:51:54.014 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-core-4.18.0-500.el8.x86_64 2/3 2023-07-13T01:51:54.706 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-4.18.0-500.el8.x86_64 3/3 2023-07-13T01:51:54.707 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-13T01:51:54.707 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-07-13T01:51:54.707 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-4.18.0-500.el8.x86_64 kernel-core-4.18.0-500.el8.x86_64 2023-07-13T01:51:54.707 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-4.18.0-500.el8.x86_64 2023-07-13T01:51:54.707 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-13T01:51:54.708 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-07-13T01:51:55.132 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2023-07-13T01:51:55.350 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-4.18.0-500.el8.x86_64 2023-07-13T01:51:55.350 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-500.el8.x86_64 2023-07-13T01:51:55.350 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-500.el8.x86_64 2023-07-13T01:51:55.350 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-07-13T01:51:55.351 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-07-13T01:51:55.351 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-07-13T01:51:55.351 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi191.front.sepia.ceph.com, path=None, version=distro) 2023-07-13T01:51:55.351 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2023-07-13T01:51:55.556 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-4.18.0-500.el8.x86_64 2023-07-13T01:51:55.557 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-500.el8.x86_64 2023-07-13T01:51:55.557 DEBUG:teuthology.orchestra.run.smithi191:> sudo rpm -qi grub2-tools 2023-07-13T01:51:55.591 INFO:teuthology.orchestra.run.smithi191.stdout:Name : grub2-tools 2023-07-13T01:51:55.591 INFO:teuthology.orchestra.run.smithi191.stdout:Epoch : 1 2023-07-13T01:51:55.591 INFO:teuthology.orchestra.run.smithi191.stdout:Version : 2.02 2023-07-13T01:51:55.592 INFO:teuthology.orchestra.run.smithi191.stdout:Release : 129.el8 2023-07-13T01:51:55.592 INFO:teuthology.orchestra.run.smithi191.stdout:Architecture: x86_64 2023-07-13T01:51:55.592 INFO:teuthology.orchestra.run.smithi191.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-07-13T01:51:55.592 INFO:teuthology.orchestra.run.smithi191.stdout:Group : System Environment/Base 2023-07-13T01:51:55.592 INFO:teuthology.orchestra.run.smithi191.stdout:Size : 9265586 2023-07-13T01:51:55.592 INFO:teuthology.orchestra.run.smithi191.stdout:License : GPLv3+ 2023-07-13T01:51:55.593 INFO:teuthology.orchestra.run.smithi191.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-07-13T01:51:55.593 INFO:teuthology.orchestra.run.smithi191.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-07-13T01:51:55.593 INFO:teuthology.orchestra.run.smithi191.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-07-13T01:51:55.593 INFO:teuthology.orchestra.run.smithi191.stdout:Build Host : kbuilder.bsys.centos.org 2023-07-13T01:51:55.593 INFO:teuthology.orchestra.run.smithi191.stdout:Relocations : (not relocatable) 2023-07-13T01:51:55.593 INFO:teuthology.orchestra.run.smithi191.stdout:Packager : CentOS BuildSystem 2023-07-13T01:51:55.593 INFO:teuthology.orchestra.run.smithi191.stdout:Vendor : CentOS 2023-07-13T01:51:55.593 INFO:teuthology.orchestra.run.smithi191.stdout:URL : http://www.gnu.org/software/grub/ 2023-07-13T01:51:55.594 INFO:teuthology.orchestra.run.smithi191.stdout:Summary : Support tools for GRUB. 2023-07-13T01:51:55.594 INFO:teuthology.orchestra.run.smithi191.stdout:Description : 2023-07-13T01:51:55.594 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-13T01:51:55.594 INFO:teuthology.orchestra.run.smithi191.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-07-13T01:51:55.594 INFO:teuthology.orchestra.run.smithi191.stdout:customizable bootloader with modular architecture. It supports a rich 2023-07-13T01:51:55.594 INFO:teuthology.orchestra.run.smithi191.stdout:variety of kernel formats, file systems, computer architectures and 2023-07-13T01:51:55.594 INFO:teuthology.orchestra.run.smithi191.stdout:hardware devices. 2023-07-13T01:51:55.594 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-13T01:51:55.595 INFO:teuthology.orchestra.run.smithi191.stdout:This subpackage provides tools for support of all platforms. 2023-07-13T01:51:55.595 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-07-13T01:51:55.596 INFO:teuthology.task.kernel:Updating grub on smithi191 to boot 4.18.0-500.el8.x86_64 2023-07-13T01:51:55.596 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-07-13T01:51:56.952 INFO:teuthology.orchestra.run.smithi191.stderr:Generating grub configuration file ... 2023-07-13T01:51:58.749 INFO:teuthology.orchestra.run.smithi191.stderr:done 2023-07-13T01:51:58.751 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2023-07-13T01:51:58.802 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.iMpKuq2oKN 2023-07-13T01:51:58.803 DEBUG:teuthology.orchestra.run.smithi191:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.iMpKuq2oKN 2023-07-13T01:51:58.854 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.iMpKuq2oKN 2023-07-13T01:51:58.955 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.iMpKuq2oKN is 6KB 2023-07-13T01:51:59.002 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.iMpKuq2oKN 2023-07-13T01:51:59.020 DEBUG:teuthology.orchestra.run.smithi191:> sudo /bin/ls /boot/loader/entries || true 2023-07-13T01:51:59.085 INFO:teuthology.orchestra.run.smithi191.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-07-13T01:51:59.086 INFO:teuthology.orchestra.run.smithi191.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-07-13T01:51:59.086 INFO:teuthology.orchestra.run.smithi191.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-07-13T01:51:59.086 INFO:teuthology.orchestra.run.smithi191.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-500.el8.x86_64.conf 2023-07-13T01:51:59.088 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-500.el8.x86_64 2023-07-13T01:51:59.258 DEBUG:teuthology.orchestra.run.smithi191:> sudo shutdown -r now 2023-07-13T01:52:10.352 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.32 2023-07-13T01:52:22.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2023-07-13T01:52:22.365 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-13T01:52:22.725 DEBUG:teuthology.orchestra.run.smithi032:> true 2023-07-13T01:52:23.017 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2023-07-13T01:52:23.017 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-500.el8.x86_64"... 2023-07-13T01:52:23.017 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2023-07-13T01:52:23.035 INFO:teuthology.orchestra.run.smithi032.stdout:4.18.0-500.el8.x86_64 2023-07-13T01:52:23.035 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-500.el8.x86_64 vs 4.18.0-500.el8.x86_64 2023-07-13T01:52:23.036 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-07-13T01:52:23.036 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-07-13T01:52:24.037 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-07-13T01:52:24.037 DEBUG:teuthology.orchestra.run.smithi032:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-13T01:52:24.091 INFO:teuthology.orchestra.run.smithi032.stdout:ttyS1 2023-07-13T01:52:24.110 DEBUG:teuthology.parallel:result is None 2023-07-13T01:52:29.261 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-07-13T01:52:29.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-07-13T01:52:29.262 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-13T01:52:47.728 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.191 2023-07-13T01:52:56.729 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-07-13T01:52:56.730 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-13T01:52:59.796 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.191 2023-07-13T01:53:11.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-07-13T01:53:11.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-13T01:53:44.523 DEBUG:teuthology.orchestra.run.smithi191:> true 2023-07-13T01:53:45.026 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-07-13T01:53:45.026 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-500.el8.x86_64"... 2023-07-13T01:53:45.026 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2023-07-13T01:53:45.043 INFO:teuthology.orchestra.run.smithi191.stdout:4.18.0-500.el8.x86_64 2023-07-13T01:53:45.043 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-500.el8.x86_64 vs 4.18.0-500.el8.x86_64 2023-07-13T01:53:45.043 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-07-13T01:53:45.044 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-07-13T01:53:46.045 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-07-13T01:53:46.045 DEBUG:teuthology.orchestra.run.smithi191:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-13T01:53:46.099 INFO:teuthology.orchestra.run.smithi191.stdout:ttyS1 2023-07-13T01:53:46.114 DEBUG:teuthology.parallel:result is None 2023-07-13T01:53:46.114 INFO:teuthology.run_tasks:Running task internal.base... 2023-07-13T01:53:46.123 INFO:teuthology.task.internal:Creating test directory... 2023-07-13T01:53:46.123 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-13T01:53:46.127 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-13T01:53:46.149 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-07-13T01:53:46.156 INFO:teuthology.run_tasks:Running task internal.archive... 2023-07-13T01:53:46.162 INFO:teuthology.task.internal:Creating archive directory... 2023-07-13T01:53:46.162 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-13T01:53:46.192 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-13T01:53:46.237 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-07-13T01:53:46.249 INFO:teuthology.task.internal:Enabling coredump saving... 2023-07-13T01:53:46.250 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-13T01:53:46.263 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-13T01:53:46.312 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-13T01:53:46.313 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-13T01:53:46.322 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-13T01:53:46.325 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-13T01:53:46.326 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-07-13T01:53:46.333 INFO:teuthology.task.internal:Configuring sudo... 2023-07-13T01:53:46.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-13T01:53:46.336 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-13T01:53:46.393 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-07-13T01:53:46.402 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-07-13T01:53:46.403 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-13T01:53:46.408 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-13T01:53:46.452 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-13T01:53:46.515 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-13T01:53:46.588 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-13T01:53:46.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-13T01:53:46.677 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2023-07-13T01:53:46.677 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-13T01:53:46.746 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-13T01:53:46.799 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-13T01:53:46.848 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-13T01:53:46.937 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-13T01:53:46.982 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-07-13T01:53:46.983 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-13T01:53:47.053 DEBUG:teuthology.orchestra.run.smithi032:> sudo service rsyslog restart 2023-07-13T01:53:47.056 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2023-07-13T01:53:47.116 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-13T01:53:47.162 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-13T01:53:47.631 INFO:teuthology.run_tasks:Running task internal.timer... 2023-07-13T01:53:47.643 INFO:teuthology.task.internal:Starting timer... 2023-07-13T01:53:47.644 INFO:teuthology.run_tasks:Running task pcp... 2023-07-13T01:53:47.658 INFO:teuthology.run_tasks:Running task selinux... 2023-07-13T01:53:47.667 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-07-13T01:53:47.668 DEBUG:teuthology.orchestra.run.smithi032:> sudo service auditd rotate 2023-07-13T01:53:47.752 INFO:teuthology.orchestra.run.smithi032.stdout:Rotating logs: 2023-07-13T01:53:47.754 DEBUG:teuthology.orchestra.run.smithi191:> sudo service auditd rotate 2023-07-13T01:53:47.841 INFO:teuthology.orchestra.run.smithi191.stdout:Rotating logs: 2023-07-13T01:53:47.843 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-07-13T01:53:47.843 DEBUG:teuthology.orchestra.run.smithi032:> /usr/sbin/getenforce 2023-07-13T01:53:47.866 INFO:teuthology.orchestra.run.smithi032.stdout:Permissive 2023-07-13T01:53:47.867 DEBUG:teuthology.orchestra.run.smithi191:> /usr/sbin/getenforce 2023-07-13T01:53:47.927 INFO:teuthology.orchestra.run.smithi191.stdout:Permissive 2023-07-13T01:53:47.927 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi032.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi191.front.sepia.ceph.com': 'permissive'} 2023-07-13T01:53:47.927 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-13T01:53:47.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-13T01:53:47.959 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-13T01:53:47.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-13T01:53:47.989 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-07-13T01:53:47.989 DEBUG:teuthology.orchestra.run.smithi032:> sudo /usr/sbin/setenforce permissive 2023-07-13T01:53:48.028 DEBUG:teuthology.orchestra.run.smithi191:> sudo /usr/sbin/setenforce permissive 2023-07-13T01:53:48.057 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-07-13T01:53:48.078 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-07-13T01:53:48.086 INFO:teuthology.repo_utils:Fetching main from origin 2023-07-13T01:53:48.223 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-07-13T01:53:48.244 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-07-13T01:53:48.246 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi032.front.sepia.ceph.com,smithi191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-07-13T02:05:59.008 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi032.front.sepia.ceph.com'), Remote(name='ubuntu@smithi191.front.sepia.ceph.com')] 2023-07-13T02:05:59.010 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2023-07-13T02:05:59.012 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-13T02:05:59.083 DEBUG:teuthology.orchestra.run.smithi032:> true 2023-07-13T02:05:59.152 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2023-07-13T02:05:59.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-07-13T02:05:59.153 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-13T02:05:59.219 DEBUG:teuthology.orchestra.run.smithi191:> true 2023-07-13T02:05:59.286 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-07-13T02:05:59.287 INFO:teuthology.run_tasks:Running task clock... 2023-07-13T02:05:59.298 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-07-13T02:05:59.299 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-13T02:05:59.300 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-13T02:05:59.302 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-13T02:05:59.302 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-13T02:05:59.337 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-13T02:05:59.353 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-13T02:05:59.360 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-13T02:05:59.374 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-13T02:05:59.424 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: ntpd: command not found 2023-07-13T02:05:59.435 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: ntpd: command not found 2023-07-13T02:05:59.437 INFO:teuthology.orchestra.run.smithi032.stdout:506 Cannot talk to daemon 2023-07-13T02:05:59.446 INFO:teuthology.orchestra.run.smithi191.stdout:506 Cannot talk to daemon 2023-07-13T02:05:59.452 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-13T02:05:59.460 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-13T02:05:59.466 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-13T02:05:59.473 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-13T02:05:59.532 INFO:teuthology.orchestra.run.smithi032.stderr:bash: ntpq: command not found 2023-07-13T02:05:59.536 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-13T02:05:59.536 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2023-07-13T02:05:59.536 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-13T02:05:59.537 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-13T02:05:59.537 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-13T02:05:59.537 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-13T02:05:59.541 INFO:teuthology.orchestra.run.smithi191.stderr:bash: ntpq: command not found 2023-07-13T02:05:59.544 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-13T02:05:59.545 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2023-07-13T02:05:59.545 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-13T02:05:59.545 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-13T02:05:59.545 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-13T02:05:59.545 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-13T02:05:59.546 INFO:teuthology.run_tasks:Running task pexec... 2023-07-13T02:05:59.556 INFO:teuthology.task.pexec:Executing custom commands... 2023-07-13T02:05:59.556 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi032.front.sepia.ceph.com 2023-07-13T02:05:59.557 DEBUG:teuthology.orchestra.run.smithi032:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-13T02:05:59.557 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi191.front.sepia.ceph.com 2023-07-13T02:05:59.557 DEBUG:teuthology.orchestra.run.smithi191:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-13T02:06:00.455 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:04:53 ago on Thu 13 Jul 2023 02:01:07 AM UTC. 2023-07-13T02:06:00.475 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:04:44 ago on Thu 13 Jul 2023 02:01:16 AM UTC. 2023-07-13T02:06:02.150 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-07-13T02:06:02.151 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-07-13T02:06:02.151 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2023-07-13T02:06:02.151 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-07-13T02:06:02.152 INFO:teuthology.orchestra.run.smithi191.stdout:Resetting modules: 2023-07-13T02:06:02.152 INFO:teuthology.orchestra.run.smithi191.stdout: container-tools 2023-07-13T02:06:02.152 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-13T02:06:02.152 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-07-13T02:06:02.152 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-07-13T02:06:02.152 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-13T02:06:02.276 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2023-07-13T02:06:02.277 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2023-07-13T02:06:02.277 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2023-07-13T02:06:02.277 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2023-07-13T02:06:02.277 INFO:teuthology.orchestra.run.smithi032.stdout:Resetting modules: 2023-07-13T02:06:02.278 INFO:teuthology.orchestra.run.smithi032.stdout: container-tools 2023-07-13T02:06:02.278 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:06:02.278 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2023-07-13T02:06:02.278 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2023-07-13T02:06:02.278 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:06:02.400 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-07-13T02:06:02.541 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2023-07-13T02:06:03.302 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:04:56 ago on Thu 13 Jul 2023 02:01:07 AM UTC. 2023-07-13T02:06:03.485 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:04:47 ago on Thu 13 Jul 2023 02:01:16 AM UTC. 2023-07-13T02:06:04.987 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-07-13T02:06:05.116 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-07-13T02:06:05.116 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2023-07-13T02:06:05.116 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-07-13T02:06:05.116 INFO:teuthology.orchestra.run.smithi191.stdout:Installing group/module packages: 2023-07-13T02:06:05.116 INFO:teuthology.orchestra.run.smithi191.stdout: aardvark-dns x86_64 2:1.6.0-1.module_el8+454+d7ef4b8d appstream 1.0 M 2023-07-13T02:06:05.117 INFO:teuthology.orchestra.run.smithi191.stdout: buildah x86_64 1:1.30.0-2.module_el8+460+8cac93c8 appstream 8.7 M 2023-07-13T02:06:05.117 INFO:teuthology.orchestra.run.smithi191.stdout: cockpit-podman noarch 70-1.module_el8+454+d7ef4b8d appstream 790 k 2023-07-13T02:06:05.117 INFO:teuthology.orchestra.run.smithi191.stdout: crun x86_64 1.8.5-1.module_el8+454+d7ef4b8d appstream 235 k 2023-07-13T02:06:05.117 INFO:teuthology.orchestra.run.smithi191.stdout: netavark x86_64 2:1.6.0-1.module_el8+454+d7ef4b8d appstream 3.6 M 2023-07-13T02:06:05.117 INFO:teuthology.orchestra.run.smithi191.stdout: python3-podman noarch 4.5.1-1.module_el8+454+d7ef4b8d appstream 177 k 2023-07-13T02:06:05.117 INFO:teuthology.orchestra.run.smithi191.stdout: skopeo x86_64 2:1.12.0-2.module_el8+460+8cac93c8 appstream 8.3 M 2023-07-13T02:06:05.117 INFO:teuthology.orchestra.run.smithi191.stdout: toolbox x86_64 0.0.99.4-1.module_el8+454+d7ef4b8d appstream 2.5 M 2023-07-13T02:06:05.118 INFO:teuthology.orchestra.run.smithi191.stdout: udica noarch 0.2.6-4.module_el8+454+d7ef4b8d appstream 52 k 2023-07-13T02:06:05.118 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2023-07-13T02:06:05.118 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-07-13T02:06:05.118 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-07-13T02:06:05.118 INFO:teuthology.orchestra.run.smithi191.stdout: yajl x86_64 2.1.0-11.el8 appstream 41 k 2023-07-13T02:06:05.118 INFO:teuthology.orchestra.run.smithi191.stdout:Installing weak dependencies: 2023-07-13T02:06:05.118 INFO:teuthology.orchestra.run.smithi191.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-07-13T02:06:05.118 INFO:teuthology.orchestra.run.smithi191.stdout:Installing module profiles: 2023-07-13T02:06:05.119 INFO:teuthology.orchestra.run.smithi191.stdout: container-tools/common 2023-07-13T02:06:05.119 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-13T02:06:05.119 INFO:teuthology.orchestra.run.smithi191.stdout:Enabling module streams: 2023-07-13T02:06:05.119 INFO:teuthology.orchestra.run.smithi191.stdout: container-tools rhel8 2023-07-13T02:06:05.119 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-13T02:06:05.119 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-07-13T02:06:05.119 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-07-13T02:06:05.119 INFO:teuthology.orchestra.run.smithi191.stdout:Install 13 Packages 2023-07-13T02:06:05.120 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-13T02:06:05.120 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 26 M 2023-07-13T02:06:05.120 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 87 M 2023-07-13T02:06:05.121 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-07-13T02:06:05.258 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2023-07-13T02:06:05.397 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2023-07-13T02:06:05.397 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2023-07-13T02:06:05.398 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2023-07-13T02:06:05.398 INFO:teuthology.orchestra.run.smithi032.stdout:Installing group/module packages: 2023-07-13T02:06:05.398 INFO:teuthology.orchestra.run.smithi032.stdout: aardvark-dns x86_64 2:1.6.0-1.module_el8+454+d7ef4b8d appstream 1.0 M 2023-07-13T02:06:05.398 INFO:teuthology.orchestra.run.smithi032.stdout: buildah x86_64 1:1.30.0-2.module_el8+460+8cac93c8 appstream 8.7 M 2023-07-13T02:06:05.398 INFO:teuthology.orchestra.run.smithi032.stdout: cockpit-podman noarch 70-1.module_el8+454+d7ef4b8d appstream 790 k 2023-07-13T02:06:05.398 INFO:teuthology.orchestra.run.smithi032.stdout: crun x86_64 1.8.5-1.module_el8+454+d7ef4b8d appstream 235 k 2023-07-13T02:06:05.399 INFO:teuthology.orchestra.run.smithi032.stdout: netavark x86_64 2:1.6.0-1.module_el8+454+d7ef4b8d appstream 3.6 M 2023-07-13T02:06:05.399 INFO:teuthology.orchestra.run.smithi032.stdout: python3-podman noarch 4.5.1-1.module_el8+454+d7ef4b8d appstream 177 k 2023-07-13T02:06:05.399 INFO:teuthology.orchestra.run.smithi032.stdout: skopeo x86_64 2:1.12.0-2.module_el8+460+8cac93c8 appstream 8.3 M 2023-07-13T02:06:05.399 INFO:teuthology.orchestra.run.smithi032.stdout: toolbox x86_64 0.0.99.4-1.module_el8+454+d7ef4b8d appstream 2.5 M 2023-07-13T02:06:05.399 INFO:teuthology.orchestra.run.smithi032.stdout: udica noarch 0.2.6-4.module_el8+454+d7ef4b8d appstream 52 k 2023-07-13T02:06:05.399 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2023-07-13T02:06:05.399 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-07-13T02:06:05.399 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-07-13T02:06:05.400 INFO:teuthology.orchestra.run.smithi032.stdout: yajl x86_64 2.1.0-11.el8 appstream 41 k 2023-07-13T02:06:05.400 INFO:teuthology.orchestra.run.smithi032.stdout:Installing weak dependencies: 2023-07-13T02:06:05.400 INFO:teuthology.orchestra.run.smithi032.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-07-13T02:06:05.400 INFO:teuthology.orchestra.run.smithi032.stdout:Installing module profiles: 2023-07-13T02:06:05.400 INFO:teuthology.orchestra.run.smithi032.stdout: container-tools/common 2023-07-13T02:06:05.400 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:06:05.400 INFO:teuthology.orchestra.run.smithi032.stdout:Enabling module streams: 2023-07-13T02:06:05.401 INFO:teuthology.orchestra.run.smithi032.stdout: container-tools rhel8 2023-07-13T02:06:05.401 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:06:05.401 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2023-07-13T02:06:05.401 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2023-07-13T02:06:05.401 INFO:teuthology.orchestra.run.smithi032.stdout:Install 13 Packages 2023-07-13T02:06:05.402 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:06:05.402 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 26 M 2023-07-13T02:06:05.402 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 87 M 2023-07-13T02:06:05.402 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2023-07-13T02:06:05.709 INFO:teuthology.orchestra.run.smithi032.stdout:[MIRROR] aardvark-dns-1.6.0-1.module_el8+454+d7ef4b8d.x86_64.rpm: Curl error (7): Couldn't connect to server for http://repos.lax.layerhost.com/centos/8-stream/AppStream/x86_64/os/Packages/aardvark-dns-1.6.0-1.module_el8%2b454%2bd7ef4b8d.x86_64.rpm [Failed to connect to repos.lax.layerhost.com port 80: No route to host] 2023-07-13T02:06:05.709 INFO:teuthology.orchestra.run.smithi032.stdout:[MIRROR] buildah-1.30.0-2.module_el8+460+8cac93c8.x86_64.rpm: Curl error (7): Couldn't connect to server for http://repos.lax.layerhost.com/centos/8-stream/AppStream/x86_64/os/Packages/buildah-1.30.0-2.module_el8%2b460%2b8cac93c8.x86_64.rpm [Failed to connect to repos.lax.layerhost.com port 80: No route to host] 2023-07-13T02:06:05.710 INFO:teuthology.orchestra.run.smithi032.stdout:[MIRROR] cockpit-podman-70-1.module_el8+454+d7ef4b8d.noarch.rpm: Curl error (7): Couldn't connect to server for http://repos.lax.layerhost.com/centos/8-stream/AppStream/x86_64/os/Packages/cockpit-podman-70-1.module_el8%2b454%2bd7ef4b8d.noarch.rpm [Failed to connect to repos.lax.layerhost.com port 80: No route to host] 2023-07-13T02:06:07.817 INFO:teuthology.orchestra.run.smithi032.stdout:(1/13): aardvark-dns-1.6.0-1.module_el8+454+d7e 448 kB/s | 1.0 MB 00:02 2023-07-13T02:06:07.942 INFO:teuthology.orchestra.run.smithi032.stdout:(2/13): cockpit-podman-70-1.module_el8+454+d7ef 335 kB/s | 790 kB 00:02 2023-07-13T02:06:08.259 INFO:teuthology.orchestra.run.smithi032.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 86 kB/s | 38 kB 00:00 2023-07-13T02:06:08.810 INFO:teuthology.orchestra.run.smithi032.stdout:(4/13): crun-1.8.5-1.module_el8+454+d7ef4b8d.x8 271 kB/s | 235 kB 00:00 2023-07-13T02:06:09.710 INFO:teuthology.orchestra.run.smithi032.stdout:(5/13): python3-podman-4.5.1-1.module_el8+454+d 197 kB/s | 177 kB 00:00 2023-07-13T02:06:10.085 INFO:teuthology.orchestra.run.smithi032.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 66 kB/s | 25 kB 00:00 2023-07-13T02:06:11.777 INFO:teuthology.orchestra.run.smithi032.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 55 kB/s | 94 kB 00:01 2023-07-13T02:06:12.444 INFO:teuthology.orchestra.run.smithi032.stdout:(8/13): buildah-1.30.0-2.module_el8+460+8cac93c 1.3 MB/s | 8.7 MB 00:06 2023-07-13T02:06:13.903 INFO:teuthology.orchestra.run.smithi032.stdout:(9/13): netavark-1.6.0-1.module_el8+454+d7ef4b8 660 kB/s | 3.6 MB 00:05 2023-07-13T02:06:14.536 INFO:teuthology.orchestra.run.smithi032.stdout:(10/13): udica-0.2.6-4.module_el8+454+d7ef4b8d. 83 kB/s | 52 kB 00:00 2023-07-13T02:06:14.970 INFO:teuthology.orchestra.run.smithi032.stdout:(11/13): yajl-2.1.0-11.el8.x86_64.rpm 94 kB/s | 41 kB 00:00 2023-07-13T02:06:20.955 INFO:teuthology.orchestra.run.smithi191.stdout:[MIRROR] aardvark-dns-1.6.0-1.module_el8+454+d7ef4b8d.x86_64.rpm: Curl error (7): Couldn't connect to server for http://repos.lax.layerhost.com/centos/8-stream/AppStream/x86_64/os/Packages/aardvark-dns-1.6.0-1.module_el8%2b454%2bd7ef4b8d.x86_64.rpm [Failed to connect to repos.lax.layerhost.com port 80: No route to host] 2023-07-13T02:06:23.328 INFO:teuthology.orchestra.run.smithi191.stdout:(1/13): aardvark-dns-1.6.0-1.module_el8+454+d7e 55 kB/s | 1.0 MB 00:18 2023-07-13T02:06:25.470 INFO:teuthology.orchestra.run.smithi032.stdout:(12/13): toolbox-0.0.99.4-1.module_el8+454+d7ef 199 kB/s | 2.5 MB 00:13 2023-07-13T02:06:34.503 INFO:teuthology.orchestra.run.smithi032.stdout:(13/13): skopeo-1.12.0-2.module_el8+460+8cac93c 372 kB/s | 8.3 MB 00:22 2023-07-13T02:06:34.504 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2023-07-13T02:06:34.504 INFO:teuthology.orchestra.run.smithi032.stdout:Total 899 kB/s | 26 MB 00:29 2023-07-13T02:06:34.778 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2023-07-13T02:06:34.800 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2023-07-13T02:06:34.800 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2023-07-13T02:06:35.069 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2023-07-13T02:06:35.072 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2023-07-13T02:06:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2023-07-13T02:06:36.021 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : yajl-2.1.0-11.el8.x86_64 1/13 2023-07-13T02:06:36.207 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-07-13T02:06:36.409 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-07-13T02:06:36.770 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-07-13T02:06:37.294 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : aardvark-dns-2:1.6.0-1.module_el8+454+d7ef4b8d.x86 5/13 2023-07-13T02:06:38.277 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : netavark-2:1.6.0-1.module_el8+454+d7ef4b8d.x86_64 6/13 2023-07-13T02:06:38.547 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : buildah-1:1.30.0-2.module_el8+460+8cac93c8.x86_64 7/13 2023-07-13T02:06:38.768 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : crun-1.8.5-1.module_el8+454+d7ef4b8d.x86_64 8/13 2023-07-13T02:06:38.892 INFO:teuthology.orchestra.run.smithi191.stdout:[MIRROR] criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64.rpm: Curl error (7): Couldn't connect to server for http://repos.lax.layerhost.com/centos/8-stream/AppStream/x86_64/os/Packages/criu-libs-3.15-3.module_el8.7.0%2b1216%2bb022c01d.x86_64.rpm [Failed to connect to repos.lax.layerhost.com port 80: No route to host] 2023-07-13T02:06:39.054 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-podman-4.5.1-1.module_el8+454+d7ef4b8d.noa 9/13 2023-07-13T02:06:39.168 INFO:teuthology.orchestra.run.smithi191.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 2.4 kB/s | 38 kB 00:15 2023-07-13T02:06:39.509 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : udica-0.2.6-4.module_el8+454+d7ef4b8d.noarch 10/13 2023-07-13T02:06:40.460 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : toolbox-0.0.99.4-1.module_el8+454+d7ef4b8d.x86_64 11/13 2023-07-13T02:06:40.699 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : skopeo-2:1.12.0-2.module_el8+460+8cac93c8.x86_64 12/13 2023-07-13T02:06:40.872 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cockpit-podman-70-1.module_el8+454+d7ef4b8d.noarch 13/13 2023-07-13T02:06:41.322 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cockpit-podman-70-1.module_el8+454+d7ef4b8d.noarch 13/13 2023-07-13T02:06:41.323 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : aardvark-dns-2:1.6.0-1.module_el8+454+d7ef4b8d.x86 1/13 2023-07-13T02:06:41.323 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : buildah-1:1.30.0-2.module_el8+460+8cac93c8.x86_64 2/13 2023-07-13T02:06:41.323 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cockpit-podman-70-1.module_el8+454+d7ef4b8d.noarch 3/13 2023-07-13T02:06:41.323 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-07-13T02:06:41.323 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : crun-1.8.5-1.module_el8+454+d7ef4b8d.x86_64 5/13 2023-07-13T02:06:41.323 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : netavark-2:1.6.0-1.module_el8+454+d7ef4b8d.x86_64 6/13 2023-07-13T02:06:41.324 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-podman-4.5.1-1.module_el8+454+d7ef4b8d.noa 7/13 2023-07-13T02:06:41.324 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-07-13T02:06:41.324 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-07-13T02:06:41.324 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : skopeo-2:1.12.0-2.module_el8+460+8cac93c8.x86_64 10/13 2023-07-13T02:06:41.324 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : toolbox-0.0.99.4-1.module_el8+454+d7ef4b8d.x86_64 11/13 2023-07-13T02:06:41.324 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : udica-0.2.6-4.module_el8+454+d7ef4b8d.noarch 12/13 2023-07-13T02:06:42.074 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : yajl-2.1.0-11.el8.x86_64 13/13 2023-07-13T02:06:42.074 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:06:42.074 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2023-07-13T02:06:42.074 INFO:teuthology.orchestra.run.smithi032.stdout: aardvark-dns-2:1.6.0-1.module_el8+454+d7ef4b8d.x86_64 2023-07-13T02:06:42.074 INFO:teuthology.orchestra.run.smithi032.stdout: buildah-1:1.30.0-2.module_el8+460+8cac93c8.x86_64 2023-07-13T02:06:42.075 INFO:teuthology.orchestra.run.smithi032.stdout: cockpit-podman-70-1.module_el8+454+d7ef4b8d.noarch 2023-07-13T02:06:42.075 INFO:teuthology.orchestra.run.smithi032.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-07-13T02:06:42.075 INFO:teuthology.orchestra.run.smithi032.stdout: crun-1.8.5-1.module_el8+454+d7ef4b8d.x86_64 2023-07-13T02:06:42.075 INFO:teuthology.orchestra.run.smithi032.stdout: netavark-2:1.6.0-1.module_el8+454+d7ef4b8d.x86_64 2023-07-13T02:06:42.075 INFO:teuthology.orchestra.run.smithi032.stdout: python3-podman-4.5.1-1.module_el8+454+d7ef4b8d.noarch 2023-07-13T02:06:42.075 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-13T02:06:42.075 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-13T02:06:42.076 INFO:teuthology.orchestra.run.smithi032.stdout: skopeo-2:1.12.0-2.module_el8+460+8cac93c8.x86_64 2023-07-13T02:06:42.076 INFO:teuthology.orchestra.run.smithi032.stdout: toolbox-0.0.99.4-1.module_el8+454+d7ef4b8d.x86_64 2023-07-13T02:06:42.076 INFO:teuthology.orchestra.run.smithi032.stdout: udica-0.2.6-4.module_el8+454+d7ef4b8d.noarch 2023-07-13T02:06:42.076 INFO:teuthology.orchestra.run.smithi032.stdout: yajl-2.1.0-11.el8.x86_64 2023-07-13T02:06:42.076 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:06:42.076 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2023-07-13T02:06:42.744 DEBUG:teuthology.parallel:result is None 2023-07-13T02:06:54.742 INFO:teuthology.orchestra.run.smithi191.stdout:[MIRROR] crun-1.8.5-1.module_el8+454+d7ef4b8d.x86_64.rpm: Curl error (7): Couldn't connect to server for http://repos.lax.layerhost.com/centos/8-stream/AppStream/x86_64/os/Packages/crun-1.8.5-1.module_el8%2b454%2bd7ef4b8d.x86_64.rpm [Failed to connect to repos.lax.layerhost.com port 80: No route to host] 2023-07-13T02:06:55.499 INFO:teuthology.orchestra.run.smithi191.stdout:(3/13): crun-1.8.5-1.module_el8+454+d7ef4b8d.x8 14 kB/s | 235 kB 00:16 2023-07-13T02:06:58.974 INFO:teuthology.orchestra.run.smithi191.stdout:(4/13): netavark-1.6.0-1.module_el8+454+d7ef4b8 1.0 MB/s | 3.6 MB 00:03 2023-07-13T02:06:59.324 INFO:teuthology.orchestra.run.smithi191.stdout:(5/13): python3-podman-4.5.1-1.module_el8+454+d 506 kB/s | 177 kB 00:00 2023-07-13T02:06:59.508 INFO:teuthology.orchestra.run.smithi191.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 135 kB/s | 25 kB 00:00 2023-07-13T02:06:59.858 INFO:teuthology.orchestra.run.smithi191.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 268 kB/s | 94 kB 00:00 2023-07-13T02:07:06.758 INFO:teuthology.orchestra.run.smithi191.stdout:(8/13): skopeo-1.12.0-2.module_el8+460+8cac93c8 1.2 MB/s | 8.3 MB 00:06 2023-07-13T02:07:09.657 INFO:teuthology.orchestra.run.smithi191.stdout:[MIRROR] buildah-1.30.0-2.module_el8+460+8cac93c8.x86_64.rpm: Curl error (7): Couldn't connect to server for http://repos.lax.layerhost.com/centos/8-stream/AppStream/x86_64/os/Packages/buildah-1.30.0-2.module_el8%2b460%2b8cac93c8.x86_64.rpm [Failed to connect to repos.lax.layerhost.com port 80: No route to host] 2023-07-13T02:07:09.658 INFO:teuthology.orchestra.run.smithi191.stdout:[MIRROR] cockpit-podman-70-1.module_el8+454+d7ef4b8d.noarch.rpm: Curl error (7): Couldn't connect to server for http://repos.lax.layerhost.com/centos/8-stream/AppStream/x86_64/os/Packages/cockpit-podman-70-1.module_el8%2b454%2bd7ef4b8d.noarch.rpm [Failed to connect to repos.lax.layerhost.com port 80: No route to host] 2023-07-13T02:07:10.682 INFO:teuthology.orchestra.run.smithi191.stdout:(9/13): cockpit-podman-70-1.module_el8+454+d7ef 12 kB/s | 790 kB 01:05 2023-07-13T02:07:10.957 INFO:teuthology.orchestra.run.smithi191.stdout:(10/13): udica-0.2.6-4.module_el8+454+d7ef4b8d. 191 kB/s | 52 kB 00:00 2023-07-13T02:07:11.174 INFO:teuthology.orchestra.run.smithi191.stdout:(11/13): yajl-2.1.0-11.el8.x86_64.rpm 188 kB/s | 41 kB 00:00 2023-07-13T02:07:13.283 INFO:teuthology.orchestra.run.smithi191.stdout:(12/13): toolbox-0.0.99.4-1.module_el8+454+d7ef 397 kB/s | 2.5 MB 00:06 2023-07-13T02:07:38.680 INFO:teuthology.orchestra.run.smithi191.stdout:(13/13): buildah-1.30.0-2.module_el8+460+8cac93 96 kB/s | 8.7 MB 01:33 2023-07-13T02:07:38.680 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-07-13T02:07:38.681 INFO:teuthology.orchestra.run.smithi191.stdout:Total 280 kB/s | 26 MB 01:33 2023-07-13T02:07:38.935 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-07-13T02:07:38.956 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-07-13T02:07:38.956 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-07-13T02:07:39.208 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-07-13T02:07:39.212 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-07-13T02:07:39.895 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-07-13T02:07:40.127 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : yajl-2.1.0-11.el8.x86_64 1/13 2023-07-13T02:07:40.306 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-07-13T02:07:40.525 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-07-13T02:07:40.773 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-07-13T02:07:41.329 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : aardvark-dns-2:1.6.0-1.module_el8+454+d7ef4b8d.x86 5/13 2023-07-13T02:07:42.255 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : netavark-2:1.6.0-1.module_el8+454+d7ef4b8d.x86_64 6/13 2023-07-13T02:07:42.501 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : buildah-1:1.30.0-2.module_el8+460+8cac93c8.x86_64 7/13 2023-07-13T02:07:42.738 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : crun-1.8.5-1.module_el8+454+d7ef4b8d.x86_64 8/13 2023-07-13T02:07:43.035 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-podman-4.5.1-1.module_el8+454+d7ef4b8d.noa 9/13 2023-07-13T02:07:43.443 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : udica-0.2.6-4.module_el8+454+d7ef4b8d.noarch 10/13 2023-07-13T02:07:44.365 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : toolbox-0.0.99.4-1.module_el8+454+d7ef4b8d.x86_64 11/13 2023-07-13T02:07:44.632 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : skopeo-2:1.12.0-2.module_el8+460+8cac93c8.x86_64 12/13 2023-07-13T02:07:44.830 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cockpit-podman-70-1.module_el8+454+d7ef4b8d.noarch 13/13 2023-07-13T02:07:45.254 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cockpit-podman-70-1.module_el8+454+d7ef4b8d.noarch 13/13 2023-07-13T02:07:45.255 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : aardvark-dns-2:1.6.0-1.module_el8+454+d7ef4b8d.x86 1/13 2023-07-13T02:07:45.255 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : buildah-1:1.30.0-2.module_el8+460+8cac93c8.x86_64 2/13 2023-07-13T02:07:45.255 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cockpit-podman-70-1.module_el8+454+d7ef4b8d.noarch 3/13 2023-07-13T02:07:45.255 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-07-13T02:07:45.255 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : crun-1.8.5-1.module_el8+454+d7ef4b8d.x86_64 5/13 2023-07-13T02:07:45.255 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : netavark-2:1.6.0-1.module_el8+454+d7ef4b8d.x86_64 6/13 2023-07-13T02:07:45.256 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-podman-4.5.1-1.module_el8+454+d7ef4b8d.noa 7/13 2023-07-13T02:07:45.256 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-07-13T02:07:45.256 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-07-13T02:07:45.256 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : skopeo-2:1.12.0-2.module_el8+460+8cac93c8.x86_64 10/13 2023-07-13T02:07:45.256 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : toolbox-0.0.99.4-1.module_el8+454+d7ef4b8d.x86_64 11/13 2023-07-13T02:07:45.256 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : udica-0.2.6-4.module_el8+454+d7ef4b8d.noarch 12/13 2023-07-13T02:07:45.939 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : yajl-2.1.0-11.el8.x86_64 13/13 2023-07-13T02:07:45.939 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-13T02:07:45.940 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-07-13T02:07:45.940 INFO:teuthology.orchestra.run.smithi191.stdout: aardvark-dns-2:1.6.0-1.module_el8+454+d7ef4b8d.x86_64 2023-07-13T02:07:45.940 INFO:teuthology.orchestra.run.smithi191.stdout: buildah-1:1.30.0-2.module_el8+460+8cac93c8.x86_64 2023-07-13T02:07:45.941 INFO:teuthology.orchestra.run.smithi191.stdout: cockpit-podman-70-1.module_el8+454+d7ef4b8d.noarch 2023-07-13T02:07:45.941 INFO:teuthology.orchestra.run.smithi191.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-07-13T02:07:45.941 INFO:teuthology.orchestra.run.smithi191.stdout: crun-1.8.5-1.module_el8+454+d7ef4b8d.x86_64 2023-07-13T02:07:45.942 INFO:teuthology.orchestra.run.smithi191.stdout: netavark-2:1.6.0-1.module_el8+454+d7ef4b8d.x86_64 2023-07-13T02:07:45.942 INFO:teuthology.orchestra.run.smithi191.stdout: python3-podman-4.5.1-1.module_el8+454+d7ef4b8d.noarch 2023-07-13T02:07:45.942 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-13T02:07:45.942 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-13T02:07:45.943 INFO:teuthology.orchestra.run.smithi191.stdout: skopeo-2:1.12.0-2.module_el8+460+8cac93c8.x86_64 2023-07-13T02:07:45.943 INFO:teuthology.orchestra.run.smithi191.stdout: toolbox-0.0.99.4-1.module_el8+454+d7ef4b8d.x86_64 2023-07-13T02:07:45.943 INFO:teuthology.orchestra.run.smithi191.stdout: udica-0.2.6-4.module_el8+454+d7ef4b8d.noarch 2023-07-13T02:07:45.944 INFO:teuthology.orchestra.run.smithi191.stdout: yajl-2.1.0-11.el8.x86_64 2023-07-13T02:07:45.944 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-13T02:07:45.944 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-07-13T02:07:46.290 DEBUG:teuthology.parallel:result is None 2023-07-13T02:07:46.290 INFO:teuthology.run_tasks:Running task nvme_loop... 2023-07-13T02:07:46.301 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2023-07-13T02:07:46.302 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2023-07-13T02:07:46.302 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/scratch_devs of=/dev/stdout 2023-07-13T02:07:46.326 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-13T02:07:46.327 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_1 2023-07-13T02:07:46.388 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-13T02:07:46.389 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-13T02:07:46.389 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 382557 Links: 1 2023-07-13T02:07:46.389 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-13T02:07:46.389 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2023-07-13T02:07:46.389 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2023-07-13 02:06:41.032190852 +0000 2023-07-13T02:07:46.389 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2023-07-13 02:02:28.155392880 +0000 2023-07-13T02:07:46.389 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2023-07-13 02:02:28.155392880 +0000 2023-07-13T02:07:46.390 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2023-07-13T02:07:46.390 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-13T02:07:46.463 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2023-07-13T02:07:46.463 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2023-07-13T02:07:46.463 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000111337 s, 4.6 MB/s 2023-07-13T02:07:46.464 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-13T02:07:46.526 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_2 2023-07-13T02:07:46.586 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-13T02:07:46.586 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-13T02:07:46.586 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 376538 Links: 1 2023-07-13T02:07:46.587 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-13T02:07:46.587 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2023-07-13T02:07:46.587 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2023-07-13 02:06:41.032190852 +0000 2023-07-13T02:07:46.587 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2023-07-13 02:02:28.684382391 +0000 2023-07-13T02:07:46.587 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2023-07-13 02:02:28.684382391 +0000 2023-07-13T02:07:46.587 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2023-07-13T02:07:46.588 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-13T02:07:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2023-07-13T02:07:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2023-07-13T02:07:46.656 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.00011418 s, 4.5 MB/s 2023-07-13T02:07:46.657 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-13T02:07:46.719 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_3 2023-07-13T02:07:46.779 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-13T02:07:46.779 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-13T02:07:46.779 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 382729 Links: 1 2023-07-13T02:07:46.779 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-13T02:07:46.779 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2023-07-13T02:07:46.779 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2023-07-13 02:06:41.033190833 +0000 2023-07-13T02:07:46.780 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2023-07-13 02:02:29.212371922 +0000 2023-07-13T02:07:46.780 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2023-07-13 02:02:29.212371922 +0000 2023-07-13T02:07:46.780 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2023-07-13T02:07:46.780 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-13T02:07:46.847 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2023-07-13T02:07:46.848 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2023-07-13T02:07:46.848 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000100379 s, 5.1 MB/s 2023-07-13T02:07:46.849 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-13T02:07:46.910 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_4 2023-07-13T02:07:46.969 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-13T02:07:46.970 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-13T02:07:46.970 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 376788 Links: 1 2023-07-13T02:07:46.970 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-13T02:07:46.970 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2023-07-13T02:07:46.971 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2023-07-13 02:06:41.033190833 +0000 2023-07-13T02:07:46.971 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2023-07-13 02:02:29.755361155 +0000 2023-07-13T02:07:46.971 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2023-07-13 02:02:29.755361155 +0000 2023-07-13T02:07:46.971 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2023-07-13T02:07:46.971 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-13T02:07:47.038 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2023-07-13T02:07:47.038 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2023-07-13T02:07:47.038 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000114106 s, 4.5 MB/s 2023-07-13T02:07:47.039 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-13T02:07:47.099 DEBUG:teuthology.orchestra.run.smithi032:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2023-07-13T02:07:47.218 INFO:teuthology.orchestra.run.smithi032.stdout:loop 2023-07-13T02:07:47.219 INFO:tasks.nvme_loop:Connecting nvme_loop smithi032:/dev/vg_nvme/lv_1... 2023-07-13T02:07:47.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2023-07-13T02:07:47.257 INFO:teuthology.orchestra.run.smithi032.stdout:1 2023-07-13T02:07:47.292 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/vg_nvme/lv_11 2023-07-13T02:07:47.332 INFO:tasks.nvme_loop:Connecting nvme_loop smithi032:/dev/vg_nvme/lv_2... 2023-07-13T02:07:47.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2023-07-13T02:07:47.408 INFO:teuthology.orchestra.run.smithi032.stdout:1 2023-07-13T02:07:47.444 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/vg_nvme/lv_21 2023-07-13T02:07:47.482 INFO:tasks.nvme_loop:Connecting nvme_loop smithi032:/dev/vg_nvme/lv_3... 2023-07-13T02:07:47.482 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2023-07-13T02:07:47.559 INFO:teuthology.orchestra.run.smithi032.stdout:1 2023-07-13T02:07:47.593 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/vg_nvme/lv_31 2023-07-13T02:07:47.632 INFO:tasks.nvme_loop:Connecting nvme_loop smithi032:/dev/vg_nvme/lv_4... 2023-07-13T02:07:47.633 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2023-07-13T02:07:47.709 INFO:teuthology.orchestra.run.smithi032.stdout:1 2023-07-13T02:07:47.744 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/vg_nvme/lv_41 2023-07-13T02:07:47.781 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2023-07-13T02:07:47.782 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/scratch_devs of=/dev/stdout 2023-07-13T02:07:47.840 DEBUG:teuthology.orchestra.run.smithi032:> sudo nvme list 2023-07-13T02:07:47.907 INFO:teuthology.orchestra.run.smithi032.stdout:Node SN Model Namespace Usage Format FW Rev 2023-07-13T02:07:47.908 INFO:teuthology.orchestra.run.smithi032.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2023-07-13T02:07:47.908 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/nvme0n1 PHMB747402DN480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2023-07-13T02:07:47.908 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/nvme1n1 f9e7fa1597c59f520ecd Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2023-07-13T02:07:47.908 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/nvme2n1 2f90e07e56aaaa9125d1 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2023-07-13T02:07:47.908 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/nvme3n1 ca8c893a11e54c7bcc94 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2023-07-13T02:07:47.909 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/nvme4n1 0dd8569261a3a5f4248e Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2023-07-13T02:07:47.909 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2023-07-13T02:07:47.910 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2023-07-13T02:07:47.910 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/scratch_devs 2023-07-13T02:07:47.977 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-07-13T02:07:47.978 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2023-07-13T02:07:47.999 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-13T02:07:48.000 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_1 2023-07-13T02:07:48.061 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-13T02:07:48.061 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-13T02:07:48.061 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 381807 Links: 1 2023-07-13T02:07:48.061 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-13T02:07:48.062 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-07-13T02:07:48.062 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-07-13 02:07:44.977858681 +0000 2023-07-13T02:07:48.062 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-07-13 02:04:17.125698992 +0000 2023-07-13T02:07:48.062 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-07-13 02:04:17.125698992 +0000 2023-07-13T02:07:48.062 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-07-13T02:07:48.062 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-13T02:07:48.131 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-07-13T02:07:48.132 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-07-13T02:07:48.132 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000251592 s, 2.0 MB/s 2023-07-13T02:07:48.133 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-13T02:07:48.193 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_2 2023-07-13T02:07:48.253 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-13T02:07:48.253 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-13T02:07:48.253 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 384210 Links: 1 2023-07-13T02:07:48.253 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-13T02:07:48.254 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-07-13T02:07:48.254 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-07-13 02:07:44.977858681 +0000 2023-07-13T02:07:48.254 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-07-13 02:04:17.670689917 +0000 2023-07-13T02:07:48.254 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-07-13 02:04:17.670689917 +0000 2023-07-13T02:07:48.254 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-07-13T02:07:48.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-13T02:07:48.320 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-07-13T02:07:48.321 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-07-13T02:07:48.321 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000219184 s, 2.3 MB/s 2023-07-13T02:07:48.322 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-13T02:07:48.382 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_3 2023-07-13T02:07:48.440 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-13T02:07:48.440 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-13T02:07:48.440 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 374516 Links: 1 2023-07-13T02:07:48.441 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-13T02:07:48.441 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-07-13T02:07:48.441 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-07-13 02:07:44.977858681 +0000 2023-07-13T02:07:48.441 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-07-13 02:04:18.207680975 +0000 2023-07-13T02:07:48.441 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-07-13 02:04:18.207680975 +0000 2023-07-13T02:07:48.441 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-07-13T02:07:48.442 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-13T02:07:48.506 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-07-13T02:07:48.506 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-07-13T02:07:48.507 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.00021174 s, 2.4 MB/s 2023-07-13T02:07:48.507 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-13T02:07:48.567 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_4 2023-07-13T02:07:48.625 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-13T02:07:48.625 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-13T02:07:48.625 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 384493 Links: 1 2023-07-13T02:07:48.625 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-13T02:07:48.626 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-07-13T02:07:48.626 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-07-13 02:07:44.977858681 +0000 2023-07-13T02:07:48.626 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-07-13 02:04:18.734672199 +0000 2023-07-13T02:07:48.626 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-07-13 02:04:18.734672199 +0000 2023-07-13T02:07:48.626 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-07-13T02:07:48.626 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-13T02:07:48.690 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-07-13T02:07:48.691 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-07-13T02:07:48.691 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000235756 s, 2.2 MB/s 2023-07-13T02:07:48.692 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-13T02:07:48.751 DEBUG:teuthology.orchestra.run.smithi191:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2023-07-13T02:07:48.874 INFO:teuthology.orchestra.run.smithi191.stdout:loop 2023-07-13T02:07:48.876 INFO:tasks.nvme_loop:Connecting nvme_loop smithi191:/dev/vg_nvme/lv_1... 2023-07-13T02:07:48.876 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2023-07-13T02:07:48.911 INFO:teuthology.orchestra.run.smithi191.stdout:1 2023-07-13T02:07:48.944 INFO:teuthology.orchestra.run.smithi191.stdout:/dev/vg_nvme/lv_11 2023-07-13T02:07:48.983 INFO:tasks.nvme_loop:Connecting nvme_loop smithi191:/dev/vg_nvme/lv_2... 2023-07-13T02:07:48.984 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2023-07-13T02:07:49.061 INFO:teuthology.orchestra.run.smithi191.stdout:1 2023-07-13T02:07:49.095 INFO:teuthology.orchestra.run.smithi191.stdout:/dev/vg_nvme/lv_21 2023-07-13T02:07:49.132 INFO:tasks.nvme_loop:Connecting nvme_loop smithi191:/dev/vg_nvme/lv_3... 2023-07-13T02:07:49.132 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2023-07-13T02:07:49.208 INFO:teuthology.orchestra.run.smithi191.stdout:1 2023-07-13T02:07:49.241 INFO:teuthology.orchestra.run.smithi191.stdout:/dev/vg_nvme/lv_31 2023-07-13T02:07:49.278 INFO:tasks.nvme_loop:Connecting nvme_loop smithi191:/dev/vg_nvme/lv_4... 2023-07-13T02:07:49.279 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2023-07-13T02:07:49.353 INFO:teuthology.orchestra.run.smithi191.stdout:1 2023-07-13T02:07:49.387 INFO:teuthology.orchestra.run.smithi191.stdout:/dev/vg_nvme/lv_41 2023-07-13T02:07:49.424 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-07-13T02:07:49.424 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2023-07-13T02:07:49.482 DEBUG:teuthology.orchestra.run.smithi191:> sudo nvme list 2023-07-13T02:07:49.548 INFO:teuthology.orchestra.run.smithi191.stdout:Node SN Model Namespace Usage Format FW Rev 2023-07-13T02:07:49.548 INFO:teuthology.orchestra.run.smithi191.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2023-07-13T02:07:49.548 INFO:teuthology.orchestra.run.smithi191.stdout:/dev/nvme0n1 CVFT6303008M400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2023-07-13T02:07:49.549 INFO:teuthology.orchestra.run.smithi191.stdout:/dev/nvme1n1 2f6b1057f20ecea2da18 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2023-07-13T02:07:49.549 INFO:teuthology.orchestra.run.smithi191.stdout:/dev/nvme2n1 8cddab037f1a86cd52ed Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2023-07-13T02:07:49.549 INFO:teuthology.orchestra.run.smithi191.stdout:/dev/nvme3n1 6a7e3a08797a3b8689fd Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2023-07-13T02:07:49.549 INFO:teuthology.orchestra.run.smithi191.stdout:/dev/nvme4n1 5be39370a9f8bdde3448 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2023-07-13T02:07:49.550 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2023-07-13T02:07:49.550 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-07-13T02:07:49.550 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/scratch_devs 2023-07-13T02:07:49.617 INFO:teuthology.run_tasks:Running task cephadm... 2023-07-13T02:07:49.706 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'global': {'mon election default strategy': 3}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '3990f91ac125621431ef1b77c2fc96421bec9e51'} 2023-07-13T02:07:49.707 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 2023-07-13T02:07:49.718 INFO:tasks.cephadm:Cluster fsid is 11418e3c-2122-11ee-9b31-001a4aab830c 2023-07-13T02:07:49.718 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-07-13T02:07:49.719 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.32', 'mon.c': '[v2:172.21.15.32:3301,v1:172.21.15.32:6790]', 'mon.b': '172.21.15.191'} 2023-07-13T02:07:49.719 INFO:tasks.cephadm:First mon is mon.a on smithi032 2023-07-13T02:07:49.719 INFO:tasks.cephadm:First mgr is y 2023-07-13T02:07:49.720 INFO:tasks.cephadm:Normalizing hostnames... 2023-07-13T02:07:49.720 DEBUG:teuthology.orchestra.run.smithi032:> sudo hostname $(hostname -s) 2023-07-13T02:07:49.756 DEBUG:teuthology.orchestra.run.smithi191:> sudo hostname $(hostname -s) 2023-07-13T02:07:49.783 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-07-13T02:07:49.784 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3990f91ac125621431ef1b77c2fc96421bec9e51 2023-07-13T02:07:50.075 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3990f91ac125621431ef1b77c2fc96421bec9e51', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/71344/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-4093-g3990f91a', 'node_name': '172.21.4.65+confusa03', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-4093.g3990f91a'}, 'url': 'https://4.chacra.ceph.com/r/ceph/cv-refactor-osd-objectstore/3990f91ac125621431ef1b77c2fc96421bec9e51/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-07-12 17:05:02.422916', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'cv-refactor-osd-objectstore', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/cv-refactor-osd-objectstore/3990f91ac125621431ef1b77c2fc96421bec9e51/centos/8/flavors/default/', 'archs': ['source', 'x86_64', 'arm64'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '3990f91ac125621431ef1b77c2fc96421bec9e51', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/71344/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-4093-g3990f91a', 'node_name': '172.21.2.6+braggi06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-4093.g3990f91a'}, 'url': 'https://1.chacra.ceph.com/r/ceph/cv-refactor-osd-objectstore/3990f91ac125621431ef1b77c2fc96421bec9e51/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-07-12 15:47:15.815282', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'cv-refactor-osd-objectstore', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/cv-refactor-osd-objectstore/3990f91ac125621431ef1b77c2fc96421bec9e51/centos/8/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2023-07-13T02:07:50.410 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref cv-refactor-osd-objectstore, sha1 3990f91ac125621431ef1b77c2fc96421bec9e51 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=3990f91ac125621431ef1b77c2fc96421bec9e51 2023-07-13T02:07:50.410 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/cv-refactor-osd-objectstore/3990f91ac125621431ef1b77c2fc96421bec9e51/centos/8/x86_64/flavors/default/cephadm 2023-07-13T02:07:50.410 DEBUG:teuthology.orchestra.run.smithi032:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/cv-refactor-osd-objectstore/3990f91ac125621431ef1b77c2fc96421bec9e51/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-13T02:07:54.853 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-rw-r--. 1 ubuntu ubuntu 648732 Jul 13 02:07 /home/ubuntu/cephtest/cephadm 2023-07-13T02:07:54.854 DEBUG:teuthology.orchestra.run.smithi191:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/cv-refactor-osd-objectstore/3990f91ac125621431ef1b77c2fc96421bec9e51/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-13T02:07:56.845 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-rw-r--. 1 ubuntu ubuntu 648732 Jul 13 02:07 /home/ubuntu/cephtest/cephadm 2023-07-13T02:07:56.845 DEBUG:teuthology.orchestra.run.smithi032:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-13T02:07:56.870 DEBUG:teuthology.orchestra.run.smithi191:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-13T02:07:56.904 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 on all hosts... 2023-07-13T02:07:56.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 pull 2023-07-13T02:07:56.912 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 pull 2023-07-13T02:07:57.700 INFO:teuthology.orchestra.run.smithi032.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51... 2023-07-13T02:07:58.576 INFO:teuthology.orchestra.run.smithi191.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51... 2023-07-13T02:08:22.669 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2023-07-13T02:08:22.670 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_version": "ceph version 18.0.0-4093-g3990f91a (3990f91ac125621431ef1b77c2fc96421bec9e51) reef (dev)", 2023-07-13T02:08:22.670 INFO:teuthology.orchestra.run.smithi191.stdout: "image_id": "73a6258f4ccfe5185d6cb87618672f07c1383d92c314b483a75698120bc3fb10", 2023-07-13T02:08:22.670 INFO:teuthology.orchestra.run.smithi191.stdout: "repo_digests": [ 2023-07-13T02:08:22.670 INFO:teuthology.orchestra.run.smithi191.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:45ecb2a0d10580384097b40b3f7ddffe0f65be8ae7a193e786b3e04c9f6ee2e9" 2023-07-13T02:08:22.670 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2023-07-13T02:08:22.671 INFO:teuthology.orchestra.run.smithi191.stdout:} 2023-07-13T02:08:25.990 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2023-07-13T02:08:25.990 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 18.0.0-4093-g3990f91a (3990f91ac125621431ef1b77c2fc96421bec9e51) reef (dev)", 2023-07-13T02:08:25.991 INFO:teuthology.orchestra.run.smithi032.stdout: "image_id": "73a6258f4ccfe5185d6cb87618672f07c1383d92c314b483a75698120bc3fb10", 2023-07-13T02:08:25.991 INFO:teuthology.orchestra.run.smithi032.stdout: "repo_digests": [ 2023-07-13T02:08:25.991 INFO:teuthology.orchestra.run.smithi032.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:45ecb2a0d10580384097b40b3f7ddffe0f65be8ae7a193e786b3e04c9f6ee2e9" 2023-07-13T02:08:25.991 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2023-07-13T02:08:25.991 INFO:teuthology.orchestra.run.smithi032.stdout:} 2023-07-13T02:08:26.009 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph 2023-07-13T02:08:26.040 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph 2023-07-13T02:08:26.078 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 777 /etc/ceph 2023-07-13T02:08:26.107 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 777 /etc/ceph 2023-07-13T02:08:26.149 INFO:tasks.cephadm:Writing seed config... 2023-07-13T02:08:26.151 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-07-13T02:08:26.152 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-07-13T02:08:26.152 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2023-07-13T02:08:26.152 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2023-07-13T02:08:26.152 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-07-13T02:08:26.153 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-07-13T02:08:26.153 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-07-13T02:08:26.153 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-07-13T02:08:26.153 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-07-13T02:08:26.154 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-07-13T02:08:26.154 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2023-07-13T02:08:26.155 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-07-13T02:08:26.173 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 11418e3c-2122-11ee-9b31-001a4aab830c mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2023-07-13T02:08:26.173 DEBUG:teuthology.orchestra.run.smithi032:mon.a> sudo journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mon.a.service 2023-07-13T02:08:26.215 DEBUG:teuthology.orchestra.run.smithi032:mgr.y> sudo journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mgr.y.service 2023-07-13T02:08:26.243 INFO:journalctl@ceph.mon.a.smithi032.stdout:-- Logs begin at Thu 2023-07-13 01:52:05 UTC. -- 2023-07-13T02:08:26.243 INFO:tasks.cephadm:Bootstrapping... 2023-07-13T02:08:26.244 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 -v bootstrap --fsid 11418e3c-2122-11ee-9b31-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.32 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-07-13T02:08:26.270 INFO:journalctl@ceph.mgr.y.smithi032.stdout:-- Logs begin at Thu 2023-07-13 01:52:05 UTC. -- 2023-07-13T02:08:26.408 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2023-07-13T02:08:26.409 INFO:teuthology.orchestra.run.smithi032.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51', '-v', 'bootstrap', '--fsid', '11418e3c-2122-11ee-9b31-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.32', '--skip-admin-label'] 2023-07-13T02:08:26.445 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stdout 4.5.1 2023-07-13T02:08:26.446 INFO:teuthology.orchestra.run.smithi032.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-07-13T02:08:26.446 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying podman|docker is present... 2023-07-13T02:08:26.481 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stdout 4.5.1 2023-07-13T02:08:26.482 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying lvm2 is present... 2023-07-13T02:08:26.482 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying time synchronization is in place... 2023-07-13T02:08:26.488 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-13T02:08:26.489 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-13T02:08:26.495 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-13T02:08:26.495 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2023-07-13T02:08:26.501 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout enabled 2023-07-13T02:08:26.506 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout active 2023-07-13T02:08:26.507 INFO:teuthology.orchestra.run.smithi032.stdout:Unit chronyd.service is enabled and running 2023-07-13T02:08:26.507 INFO:teuthology.orchestra.run.smithi032.stdout:Repeating the final host check... 2023-07-13T02:08:26.543 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stdout 4.5.1 2023-07-13T02:08:26.544 INFO:teuthology.orchestra.run.smithi032.stdout:podman (/bin/podman) version 4.5.1 is present 2023-07-13T02:08:26.544 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl is present 2023-07-13T02:08:26.544 INFO:teuthology.orchestra.run.smithi032.stdout:lvcreate is present 2023-07-13T02:08:26.550 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-13T02:08:26.550 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-13T02:08:26.556 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-13T02:08:26.556 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2023-07-13T02:08:26.562 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout enabled 2023-07-13T02:08:26.568 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout active 2023-07-13T02:08:26.568 INFO:teuthology.orchestra.run.smithi032.stdout:Unit chronyd.service is enabled and running 2023-07-13T02:08:26.568 INFO:teuthology.orchestra.run.smithi032.stdout:Host looks OK 2023-07-13T02:08:26.568 INFO:teuthology.orchestra.run.smithi032.stdout:Cluster fsid: 11418e3c-2122-11ee-9b31-001a4aab830c 2023-07-13T02:08:26.569 INFO:teuthology.orchestra.run.smithi032.stdout:Acquiring lock 140156119104368 on /run/cephadm/11418e3c-2122-11ee-9b31-001a4aab830c.lock 2023-07-13T02:08:26.569 INFO:teuthology.orchestra.run.smithi032.stdout:Lock 140156119104368 acquired on /run/cephadm/11418e3c-2122-11ee-9b31-001a4aab830c.lock 2023-07-13T02:08:26.569 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying IP 172.21.15.32 port 3300 ... 2023-07-13T02:08:26.570 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying IP 172.21.15.32 port 6789 ... 2023-07-13T02:08:26.570 INFO:teuthology.orchestra.run.smithi032.stdout:Base mon IP(s) is [172.21.15.32:3300, 172.21.15.32:6789], mon addrv is [v2:172.21.15.32:3300,v1:172.21.15.32:6789] 2023-07-13T02:08:26.573 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.32 metric 100 2023-07-13T02:08:26.573 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.32 metric 100 2023-07-13T02:08:26.576 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-07-13T02:08:26.576 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-07-13T02:08:26.576 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1795sec hoplimit 64 pref medium 2023-07-13T02:08:26.579 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-07-13T02:08:26.579 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-07-13T02:08:26.579 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-13T02:08:26.579 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2023-07-13T02:08:26.579 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:2602/64 scope link 2023-07-13T02:08:26.579 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-13T02:08:26.580 INFO:teuthology.orchestra.run.smithi032.stdout:Mon IP `172.21.15.32` is in CIDR network `172.21.0.0/20` 2023-07-13T02:08:26.580 INFO:teuthology.orchestra.run.smithi032.stdout:Mon IP `172.21.15.32` is in CIDR network `172.21.0.0/20` 2023-07-13T02:08:26.580 INFO:teuthology.orchestra.run.smithi032.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-07-13T02:08:26.581 INFO:teuthology.orchestra.run.smithi032.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-07-13T02:08:26.581 INFO:teuthology.orchestra.run.smithi032.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51... 2023-07-13T02:08:26.935 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stdout 73a6258f4ccfe5185d6cb87618672f07c1383d92c314b483a75698120bc3fb10 2023-07-13T02:08:26.936 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51... 2023-07-13T02:08:26.936 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Getting image source signatures 2023-07-13T02:08:26.936 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Copying blob sha256:7cfd9bdfa43af51cf862ff45df0fe4cbedd46d22cfbeed896d4c6dc774ab796c 2023-07-13T02:08:26.936 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Copying blob sha256:6022c611d474f2b306ddd1edfc8360123ece5266119096f25fde843d7967e636 2023-07-13T02:08:26.937 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Copying config sha256:73a6258f4ccfe5185d6cb87618672f07c1383d92c314b483a75698120bc3fb10 2023-07-13T02:08:26.937 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Writing manifest to image destination 2023-07-13T02:08:26.937 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Storing signatures 2023-07-13T02:08:27.545 INFO:teuthology.orchestra.run.smithi032.stdout:ceph: stdout ceph version 18.0.0-4093-g3990f91a (3990f91ac125621431ef1b77c2fc96421bec9e51) reef (dev) 2023-07-13T02:08:27.545 INFO:teuthology.orchestra.run.smithi032.stdout:Ceph version: ceph version 18.0.0-4093-g3990f91a (3990f91ac125621431ef1b77c2fc96421bec9e51) reef (dev) 2023-07-13T02:08:27.545 INFO:teuthology.orchestra.run.smithi032.stdout:Extracting ceph user uid/gid from container image... 2023-07-13T02:08:28.185 INFO:teuthology.orchestra.run.smithi032.stdout:stat: stdout 167 167 2023-07-13T02:08:28.185 INFO:teuthology.orchestra.run.smithi032.stdout:Creating initial keys... 2023-07-13T02:08:28.702 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-authtool: stdout AQCcXK9kUwXSFRAA/NihmEVwJjEYwSt7yadmJA== 2023-07-13T02:08:29.235 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-authtool: stdout AQCcXK9keedZMxAAnwY58csKP2CJ0XZAwd/fyQ== 2023-07-13T02:08:29.809 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-authtool: stdout AQCdXK9ksaY9HBAAZeIWZ1BtRunCTZKt6AIVNw== 2023-07-13T02:08:29.809 INFO:teuthology.orchestra.run.smithi032.stdout:Creating initial monmap... 2023-07-13T02:08:30.362 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-13T02:08:30.362 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-07-13T02:08:30.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 11418e3c-2122-11ee-9b31-001a4aab830c 2023-07-13T02:08:30.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-13T02:08:30.363 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool for a [v2:172.21.15.32:3300,v1:172.21.15.32:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-13T02:08:30.363 INFO:teuthology.orchestra.run.smithi032.stdout:setting min_mon_release = pacific 2023-07-13T02:08:30.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: set fsid to 11418e3c-2122-11ee-9b31-001a4aab830c 2023-07-13T02:08:30.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-13T02:08:30.363 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:08:30.364 INFO:teuthology.orchestra.run.smithi032.stdout:Creating mon... 2023-07-13T02:08:30.881 INFO:teuthology.orchestra.run.smithi032.stdout:create mon.a on 2023-07-13T02:08:31.079 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-07-13T02:08:31.177 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-11418e3c-2122-11ee-9b31-001a4aab830c.target → /etc/systemd/system/ceph-11418e3c-2122-11ee-9b31-001a4aab830c.target. 2023-07-13T02:08:31.177 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-11418e3c-2122-11ee-9b31-001a4aab830c.target → /etc/systemd/system/ceph-11418e3c-2122-11ee-9b31-001a4aab830c.target. 2023-07-13T02:08:31.317 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mon.a 2023-07-13T02:08:31.317 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to reset failed state of unit ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mon.a.service: Unit ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mon.a.service not loaded. 2023-07-13T02:08:31.418 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-11418e3c-2122-11ee-9b31-001a4aab830c.target.wants/ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-11418e3c-2122-11ee-9b31-001a4aab830c@.service. 2023-07-13T02:08:31.796 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-13T02:08:31.796 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2023-07-13T02:08:31.805 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-13T02:08:31.805 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2023-07-13T02:08:31.805 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2023-07-13T02:08:31.805 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to enable service . firewalld.service is not available 2023-07-13T02:08:31.805 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mon to start... 2023-07-13T02:08:31.806 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mon... 2023-07-13T02:08:32.135 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:31 smithi032 ceph-mon[104476]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-13T02:08:32.517 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout cluster: 2023-07-13T02:08:32.517 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout id: 11418e3c-2122-11ee-9b31-001a4aab830c 2023-07-13T02:08:32.517 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-07-13T02:08:32.518 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2023-07-13T02:08:32.518 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout services: 2023-07-13T02:08:32.518 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.396644s) 2023-07-13T02:08:32.518 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-07-13T02:08:32.518 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-07-13T02:08:32.518 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2023-07-13T02:08:32.518 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout data: 2023-07-13T02:08:32.519 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-07-13T02:08:32.519 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-07-13T02:08:32.519 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-07-13T02:08:32.519 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout pgs: 2023-07-13T02:08:32.519 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2023-07-13T02:08:32.519 INFO:teuthology.orchestra.run.smithi032.stdout:mon is available 2023-07-13T02:08:32.519 INFO:teuthology.orchestra.run.smithi032.stdout:Assimilating anything we can from ceph.conf... 2023-07-13T02:08:32.910 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:32 smithi032 ceph-mon[104476]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-13T02:08:32.911 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:32 smithi032 ceph-mon[104476]: monmap e1: 1 mons at {a=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0]} removed_ranks: {} 2023-07-13T02:08:32.911 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:32 smithi032 ceph-mon[104476]: fsmap 2023-07-13T02:08:32.911 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:32 smithi032 ceph-mon[104476]: osdmap e1: 0 total, 0 up, 0 in 2023-07-13T02:08:32.912 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:32 smithi032 ceph-mon[104476]: mgrmap e1: no daemons active 2023-07-13T02:08:32.912 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:32 smithi032 ceph-mon[104476]: from='client.? 172.21.15.32:0/3934702177' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-07-13T02:08:33.243 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2023-07-13T02:08:33.243 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [global] 2023-07-13T02:08:33.243 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout fsid = 11418e3c-2122-11ee-9b31-001a4aab830c 2023-07-13T02:08:33.243 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.32:3300,v1:172.21.15.32:6789] 2023-07-13T02:08:33.243 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-13T02:08:33.244 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-13T02:08:33.244 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-13T02:08:33.244 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-13T02:08:33.244 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2023-07-13T02:08:33.244 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-13T02:08:33.245 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = False 2023-07-13T02:08:33.245 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-13T02:08:33.245 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2023-07-13T02:08:33.245 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [osd] 2023-07-13T02:08:33.245 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-13T02:08:33.245 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-13T02:08:33.245 INFO:teuthology.orchestra.run.smithi032.stdout:Generating new minimal ceph.conf... 2023-07-13T02:08:33.885 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:33 smithi032 ceph-mon[104476]: from='client.? 172.21.15.32:0/1845941787' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-13T02:08:33.885 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:33 smithi032 ceph-mon[104476]: from='client.? 172.21.15.32:0/1845941787' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-07-13T02:08:33.885 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:33 smithi032 ceph-mon[104476]: from='client.? 172.21.15.32:0/3292780349' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:08:33.933 INFO:teuthology.orchestra.run.smithi032.stdout:Restarting the monitor... 2023-07-13T02:08:34.385 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:33 smithi032 systemd[1]: Stopping Ceph mon.a for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:08:34.385 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:34 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mon-a[104472]: 2023-07-13T02:08:34.110+0000 7f45402fb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-13T02:08:34.385 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:34 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mon-a[104472]: 2023-07-13T02:08:34.110+0000 7f45402fb700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-07-13T02:08:34.698 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:34 smithi032 bash[104729]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mon-a 2023-07-13T02:08:34.963 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:34 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mon.a.service: Succeeded. 2023-07-13T02:08:34.964 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:34 smithi032 systemd[1]: Stopped Ceph mon.a for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:08:34.964 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:34 smithi032 systemd[1]: Starting Ceph mon.a for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:08:35.066 INFO:teuthology.orchestra.run.smithi032.stdout:Setting mon public_network to 172.21.0.0/20 2023-07-13T02:08:35.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: set uid:gid to 167:167 (ceph:ceph) 2023-07-13T02:08:35.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: ceph version 18.0.0-4093-g3990f91a (3990f91ac125621431ef1b77c2fc96421bec9e51) reef (dev), process ceph-mon, pid 2 2023-07-13T02:08:35.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: pidfile_write: ignore empty --pid-file 2023-07-13T02:08:35.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: load: jerasure load: lrc 2023-07-13T02:08:35.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: RocksDB version: 7.9.2 2023-07-13T02:08:35.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Git sha 0 2023-07-13T02:08:35.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Compile date 2023-07-12 14:33:48 2023-07-13T02:08:35.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: DB SUMMARY 2023-07-13T02:08:35.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: DB Session ID: D1HLT2KQ7MVDMQ66CZKU 2023-07-13T02:08:35.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: CURRENT file: CURRENT 2023-07-13T02:08:35.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: IDENTITY file: IDENTITY 2023-07-13T02:08:35.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-07-13T02:08:35.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-07-13T02:08:35.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 83304 ; 2023-07-13T02:08:35.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.error_if_exists: 0 2023-07-13T02:08:35.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.create_if_missing: 0 2023-07-13T02:08:35.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.paranoid_checks: 1 2023-07-13T02:08:35.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-13T02:08:35.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-13T02:08:35.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-13T02:08:35.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.env: 0x55d88813e720 2023-07-13T02:08:35.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.fs: PosixFileSystem 2023-07-13T02:08:35.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.info_log: 0x55d88b2c13c0 2023-07-13T02:08:35.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_file_opening_threads: 16 2023-07-13T02:08:35.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.statistics: (nil) 2023-07-13T02:08:35.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.use_fsync: 0 2023-07-13T02:08:35.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_log_file_size: 0 2023-07-13T02:08:35.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-13T02:08:35.395 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-13T02:08:35.395 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.keep_log_file_num: 1000 2023-07-13T02:08:35.395 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.recycle_log_file_num: 0 2023-07-13T02:08:35.395 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.allow_fallocate: 1 2023-07-13T02:08:35.396 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.allow_mmap_reads: 0 2023-07-13T02:08:35.396 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.allow_mmap_writes: 0 2023-07-13T02:08:35.396 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.use_direct_reads: 0 2023-07-13T02:08:35.397 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-13T02:08:35.397 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.create_missing_column_families: 0 2023-07-13T02:08:35.397 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.db_log_dir: 2023-07-13T02:08:35.397 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.wal_dir: 2023-07-13T02:08:35.398 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-13T02:08:35.398 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-13T02:08:35.398 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-13T02:08:35.398 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-13T02:08:35.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-13T02:08:35.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-13T02:08:35.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.advise_random_on_open: 1 2023-07-13T02:08:35.400 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.db_write_buffer_size: 0 2023-07-13T02:08:35.400 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.write_buffer_manager: 0x55d88a5323c0 2023-07-13T02:08:35.400 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-13T02:08:35.400 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-13T02:08:35.401 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-13T02:08:35.401 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.rate_limiter: (nil) 2023-07-13T02:08:35.401 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-13T02:08:35.402 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.wal_recovery_mode: 2 2023-07-13T02:08:35.402 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.enable_thread_tracking: 0 2023-07-13T02:08:35.402 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.enable_pipelined_write: 0 2023-07-13T02:08:35.402 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.unordered_write: 0 2023-07-13T02:08:35.403 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-13T02:08:35.403 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-13T02:08:35.403 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-13T02:08:35.403 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-13T02:08:35.403 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.row_cache: None 2023-07-13T02:08:35.403 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.wal_filter: None 2023-07-13T02:08:35.403 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-13T02:08:35.403 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.allow_ingest_behind: 0 2023-07-13T02:08:35.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.two_write_queues: 0 2023-07-13T02:08:35.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.manual_wal_flush: 0 2023-07-13T02:08:35.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.wal_compression: 0 2023-07-13T02:08:35.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.atomic_flush: 0 2023-07-13T02:08:35.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-13T02:08:35.405 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-13T02:08:35.405 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-13T02:08:35.405 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.log_readahead_size: 0 2023-07-13T02:08:35.405 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-13T02:08:35.405 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.best_efforts_recovery: 0 2023-07-13T02:08:35.405 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-13T02:08:35.405 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-13T02:08:35.405 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.allow_data_in_errors: 0 2023-07-13T02:08:35.406 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.db_host_id: __hostname__ 2023-07-13T02:08:35.406 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-13T02:08:35.406 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_background_jobs: 2 2023-07-13T02:08:35.406 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_background_compactions: -1 2023-07-13T02:08:35.406 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_subcompactions: 1 2023-07-13T02:08:35.406 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-13T02:08:35.406 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-13T02:08:35.407 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-13T02:08:35.407 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_total_wal_size: 0 2023-07-13T02:08:35.407 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-13T02:08:35.407 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-13T02:08:35.407 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-13T02:08:35.407 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-13T02:08:35.407 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_open_files: -1 2023-07-13T02:08:35.408 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.bytes_per_sync: 0 2023-07-13T02:08:35.408 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-13T02:08:35.408 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-13T02:08:35.408 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.compaction_readahead_size: 0 2023-07-13T02:08:35.408 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_background_flushes: -1 2023-07-13T02:08:35.408 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Compression algorithms supported: 2023-07-13T02:08:35.408 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-13T02:08:35.409 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: kZSTD supported: 0 2023-07-13T02:08:35.409 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: kXpressCompression supported: 0 2023-07-13T02:08:35.409 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: kLZ4HCCompression supported: 1 2023-07-13T02:08:35.409 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: kZlibCompression supported: 1 2023-07-13T02:08:35.409 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: kSnappyCompression supported: 1 2023-07-13T02:08:35.409 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: kLZ4Compression supported: 1 2023-07-13T02:08:35.409 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: kBZip2Compression supported: 0 2023-07-13T02:08:35.410 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-13T02:08:35.410 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-13T02:08:35.410 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-07-13T02:08:35.410 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-13T02:08:35.410 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-13T02:08:35.410 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.merge_operator: 2023-07-13T02:08:35.410 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.compaction_filter: None 2023-07-13T02:08:35.411 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.compaction_filter_factory: None 2023-07-13T02:08:35.411 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.sst_partitioner_factory: None 2023-07-13T02:08:35.411 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-13T02:08:35.411 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-13T02:08:35.411 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d88b2c14e0) 2023-07-13T02:08:35.411 INFO:journalctl@ceph.mon.a.smithi032.stdout: cache_index_and_filter_blocks: 1 2023-07-13T02:08:35.411 INFO:journalctl@ceph.mon.a.smithi032.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-13T02:08:35.411 INFO:journalctl@ceph.mon.a.smithi032.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-13T02:08:35.412 INFO:journalctl@ceph.mon.a.smithi032.stdout: pin_top_level_index_and_filter: 1 2023-07-13T02:08:35.412 INFO:journalctl@ceph.mon.a.smithi032.stdout: index_type: 0 2023-07-13T02:08:35.412 INFO:journalctl@ceph.mon.a.smithi032.stdout: data_block_index_type: 0 2023-07-13T02:08:35.412 INFO:journalctl@ceph.mon.a.smithi032.stdout: index_shortening: 1 2023-07-13T02:08:35.412 INFO:journalctl@ceph.mon.a.smithi032.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-13T02:08:35.412 INFO:journalctl@ceph.mon.a.smithi032.stdout: checksum: 4 2023-07-13T02:08:35.412 INFO:journalctl@ceph.mon.a.smithi032.stdout: no_block_cache: 0 2023-07-13T02:08:35.413 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_cache: 0x55d88a5d31f0 2023-07-13T02:08:35.413 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_cache_name: BinnedLRUCache 2023-07-13T02:08:35.413 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_cache_options: 2023-07-13T02:08:35.413 INFO:journalctl@ceph.mon.a.smithi032.stdout: capacity : 536870912 2023-07-13T02:08:35.413 INFO:journalctl@ceph.mon.a.smithi032.stdout: num_shard_bits : 4 2023-07-13T02:08:35.413 INFO:journalctl@ceph.mon.a.smithi032.stdout: strict_capacity_limit : 0 2023-07-13T02:08:35.413 INFO:journalctl@ceph.mon.a.smithi032.stdout: high_pri_pool_ratio: 0.000 2023-07-13T02:08:35.413 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_cache_compressed: (nil) 2023-07-13T02:08:35.414 INFO:journalctl@ceph.mon.a.smithi032.stdout: persistent_cache: (nil) 2023-07-13T02:08:35.414 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_size: 4096 2023-07-13T02:08:35.414 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_size_deviation: 10 2023-07-13T02:08:35.414 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_restart_interval: 16 2023-07-13T02:08:35.414 INFO:journalctl@ceph.mon.a.smithi032.stdout: index_block_restart_interval: 1 2023-07-13T02:08:35.414 INFO:journalctl@ceph.mon.a.smithi032.stdout: metadata_block_size: 4096 2023-07-13T02:08:35.414 INFO:journalctl@ceph.mon.a.smithi032.stdout: partition_filters: 0 2023-07-13T02:08:35.415 INFO:journalctl@ceph.mon.a.smithi032.stdout: use_delta_encoding: 1 2023-07-13T02:08:35.415 INFO:journalctl@ceph.mon.a.smithi032.stdout: filter_policy: bloomfilter 2023-07-13T02:08:35.415 INFO:journalctl@ceph.mon.a.smithi032.stdout: whole_key_filtering: 1 2023-07-13T02:08:35.415 INFO:journalctl@ceph.mon.a.smithi032.stdout: verify_compression: 0 2023-07-13T02:08:35.415 INFO:journalctl@ceph.mon.a.smithi032.stdout: read_amp_bytes_per_bit: 0 2023-07-13T02:08:35.415 INFO:journalctl@ceph.mon.a.smithi032.stdout: format_version: 5 2023-07-13T02:08:35.415 INFO:journalctl@ceph.mon.a.smithi032.stdout: enable_index_compression: 1 2023-07-13T02:08:35.416 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_align: 0 2023-07-13T02:08:35.416 INFO:journalctl@ceph.mon.a.smithi032.stdout: max_auto_readahead_size: 262144 2023-07-13T02:08:35.416 INFO:journalctl@ceph.mon.a.smithi032.stdout: prepopulate_block_cache: 0 2023-07-13T02:08:35.416 INFO:journalctl@ceph.mon.a.smithi032.stdout: initial_auto_readahead_size: 8192 2023-07-13T02:08:35.416 INFO:journalctl@ceph.mon.a.smithi032.stdout: num_file_reads_for_auto_readahead: 2 2023-07-13T02:08:35.417 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.write_buffer_size: 33554432 2023-07-13T02:08:35.417 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_write_buffer_number: 2 2023-07-13T02:08:35.417 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.compression: NoCompression 2023-07-13T02:08:35.417 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.bottommost_compression: Disabled 2023-07-13T02:08:35.417 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.prefix_extractor: nullptr 2023-07-13T02:08:35.417 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-13T02:08:35.418 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.num_levels: 7 2023-07-13T02:08:35.418 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-13T02:08:35.418 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-13T02:08:35.418 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-13T02:08:35.418 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-13T02:08:35.418 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-13T02:08:35.419 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-13T02:08:35.419 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-13T02:08:35.419 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-13T02:08:35.419 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-13T02:08:35.419 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-13T02:08:35.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-13T02:08:35.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-13T02:08:35.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-13T02:08:35.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.compression_opts.level: 32767 2023-07-13T02:08:35.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.compression_opts.strategy: 0 2023-07-13T02:08:35.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-13T02:08:35.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-13T02:08:35.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-13T02:08:35.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-13T02:08:35.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.compression_opts.enabled: false 2023-07-13T02:08:35.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-13T02:08:35.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-13T02:08:35.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-13T02:08:35.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-13T02:08:35.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.target_file_size_base: 67108864 2023-07-13T02:08:35.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-13T02:08:35.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-13T02:08:35.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-13T02:08:35.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-13T02:08:35.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-13T02:08:35.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-13T02:08:35.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-13T02:08:35.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-13T02:08:35.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-13T02:08:35.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-13T02:08:35.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-13T02:08:35.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-13T02:08:35.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-13T02:08:35.424 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-13T02:08:35.424 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.arena_block_size: 1048576 2023-07-13T02:08:35.424 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-13T02:08:35.424 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-13T02:08:35.424 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.disable_auto_compactions: 0 2023-07-13T02:08:35.424 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-13T02:08:35.425 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-13T02:08:35.425 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-13T02:08:35.425 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-13T02:08:35.425 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-13T02:08:35.425 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-13T02:08:35.425 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-13T02:08:35.425 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-13T02:08:35.426 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-13T02:08:35.426 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-13T02:08:35.426 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.table_properties_collectors: 2023-07-13T02:08:35.426 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.inplace_update_support: 0 2023-07-13T02:08:35.426 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-13T02:08:35.426 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-13T02:08:35.426 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-13T02:08:35.427 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-13T02:08:35.427 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.bloom_locality: 0 2023-07-13T02:08:35.427 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.max_successive_merges: 0 2023-07-13T02:08:35.427 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-13T02:08:35.427 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.paranoid_file_checks: 0 2023-07-13T02:08:35.427 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.force_consistency_checks: 1 2023-07-13T02:08:35.428 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.report_bg_io_stats: 0 2023-07-13T02:08:35.428 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.ttl: 2592000 2023-07-13T02:08:35.428 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-13T02:08:35.428 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-13T02:08:35.428 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-13T02:08:35.428 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.enable_blob_files: false 2023-07-13T02:08:35.428 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.min_blob_size: 0 2023-07-13T02:08:35.429 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.blob_file_size: 268435456 2023-07-13T02:08:35.429 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-13T02:08:35.429 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-13T02:08:35.429 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-13T02:08:35.429 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-13T02:08:35.429 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-13T02:08:35.430 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.blob_file_starting_level: 0 2023-07-13T02:08:35.430 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-13T02:08:35.430 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-07-13T02:08:35.430 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-07-13T02:08:35.430 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: cc31c7bd-3e9d-4838-91f3-6bffe1a69939 2023-07-13T02:08:35.430 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689214115029831, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-07-13T02:08:35.430 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-07-13T02:08:35.430 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689214115030759, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 80198, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 244, "table_properties": {"data_size": 78385, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9867, "raw_average_key_size": 47, "raw_value_size": 72719, "raw_average_value_size": 351, "num_data_blocks": 10, "num_entries": 207, "num_filter_entries": 207, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689214115, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "cc31c7bd-3e9d-4838-91f3-6bffe1a69939", "db_session_id": "D1HLT2KQ7MVDMQ66CZKU", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-07-13T02:08:35.431 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689214115030838, "job": 1, "event": "recovery_finished"} 2023-07-13T02:08:35.431 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-07-13T02:08:35.431 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-13T02:08:35.431 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d88a670000 2023-07-13T02:08:35.431 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: DB pointer 0x55d88a65c000 2023-07-13T02:08:35.431 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-13T02:08:35.431 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-13T02:08:35.432 INFO:journalctl@ceph.mon.a.smithi032.stdout: ** DB Stats ** 2023-07-13T02:08:35.432 INFO:journalctl@ceph.mon.a.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-13T02:08:35.432 INFO:journalctl@ceph.mon.a.smithi032.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-13T02:08:35.432 INFO:journalctl@ceph.mon.a.smithi032.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-13T02:08:35.432 INFO:journalctl@ceph.mon.a.smithi032.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-13T02:08:35.432 INFO:journalctl@ceph.mon.a.smithi032.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-13T02:08:35.432 INFO:journalctl@ceph.mon.a.smithi032.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-13T02:08:35.433 INFO:journalctl@ceph.mon.a.smithi032.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-13T02:08:35.433 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2023-07-13T02:08:35.433 INFO:journalctl@ceph.mon.a.smithi032.stdout: ** Compaction Stats [default] ** 2023-07-13T02:08:35.433 INFO:journalctl@ceph.mon.a.smithi032.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-13T02:08:35.433 INFO:journalctl@ceph.mon.a.smithi032.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-13T02:08:35.433 INFO:journalctl@ceph.mon.a.smithi032.stdout: L0 2/0 80.15 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 130.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-13T02:08:35.433 INFO:journalctl@ceph.mon.a.smithi032.stdout: Sum 2/0 80.15 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 130.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-13T02:08:35.434 INFO:journalctl@ceph.mon.a.smithi032.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 130.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-13T02:08:35.434 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2023-07-13T02:08:35.434 INFO:journalctl@ceph.mon.a.smithi032.stdout: ** Compaction Stats [default] ** 2023-07-13T02:08:35.434 INFO:journalctl@ceph.mon.a.smithi032.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-13T02:08:35.434 INFO:journalctl@ceph.mon.a.smithi032.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-13T02:08:35.434 INFO:journalctl@ceph.mon.a.smithi032.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 130.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-13T02:08:35.434 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2023-07-13T02:08:35.435 INFO:journalctl@ceph.mon.a.smithi032.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-13T02:08:35.435 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2023-07-13T02:08:35.435 INFO:journalctl@ceph.mon.a.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-13T02:08:35.435 INFO:journalctl@ceph.mon.a.smithi032.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-13T02:08:35.435 INFO:journalctl@ceph.mon.a.smithi032.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-13T02:08:35.435 INFO:journalctl@ceph.mon.a.smithi032.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-13T02:08:35.435 INFO:journalctl@ceph.mon.a.smithi032.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-13T02:08:35.436 INFO:journalctl@ceph.mon.a.smithi032.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-13T02:08:35.436 INFO:journalctl@ceph.mon.a.smithi032.stdout: Cumulative compaction: 0.00 GB write, 25.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-13T02:08:35.436 INFO:journalctl@ceph.mon.a.smithi032.stdout: Interval compaction: 0.00 GB write, 25.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-13T02:08:35.436 INFO:journalctl@ceph.mon.a.smithi032.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-13T02:08:35.436 INFO:journalctl@ceph.mon.a.smithi032.stdout: Block cache BinnedLRUCache@0x55d88a5d31f0#2 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-07-13T02:08:35.436 INFO:journalctl@ceph.mon.a.smithi032.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2023-07-13T02:08:35.437 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2023-07-13T02:08:35.437 INFO:journalctl@ceph.mon.a.smithi032.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-13T02:08:35.437 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: starting mon.a rank 0 at public addrs [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] at bind addrs [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 11418e3c-2122-11ee-9b31-001a4aab830c 2023-07-13T02:08:35.437 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: mon.a@-1(???) e1 preinit fsid 11418e3c-2122-11ee-9b31-001a4aab830c 2023-07-13T02:08:35.437 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: mon.a@-1(???).mds e1 new map 2023-07-13T02:08:35.437 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: mon.a@-1(???).mds e1 print_map 2023-07-13T02:08:35.437 INFO:journalctl@ceph.mon.a.smithi032.stdout: e1 2023-07-13T02:08:35.437 INFO:journalctl@ceph.mon.a.smithi032.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-13T02:08:35.438 INFO:journalctl@ceph.mon.a.smithi032.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-13T02:08:35.438 INFO:journalctl@ceph.mon.a.smithi032.stdout: legacy client fscid: -1 2023-07-13T02:08:35.438 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2023-07-13T02:08:35.438 INFO:journalctl@ceph.mon.a.smithi032.stdout: No filesystems configured 2023-07-13T02:08:35.438 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-13T02:08:35.438 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-13T02:08:35.438 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-13T02:08:35.439 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-13T02:08:35.439 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-07-13T02:08:35.439 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: expand_channel_meta expand map: {default=false} 2023-07-13T02:08:35.439 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: expand_channel_meta from 'false' to 'false' 2023-07-13T02:08:35.439 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: expand_channel_meta expanded map: {default=false} 2023-07-13T02:08:35.439 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: expand_channel_meta expand map: {default=info} 2023-07-13T02:08:35.440 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: expand_channel_meta from 'info' to 'info' 2023-07-13T02:08:35.440 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: expand_channel_meta expanded map: {default=info} 2023-07-13T02:08:35.440 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: expand_channel_meta expand map: {default=daemon} 2023-07-13T02:08:35.440 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-13T02:08:35.440 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: expand_channel_meta expanded map: {default=daemon} 2023-07-13T02:08:35.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: expand_channel_meta expand map: {default=debug} 2023-07-13T02:08:35.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: expand_channel_meta from 'debug' to 'debug' 2023-07-13T02:08:35.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: expand_channel_meta expanded map: {default=debug} 2023-07-13T02:08:35.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 bash[104839]: 629ca0477cbade7c179cc6fd2e9cedd2c83cb2bd464962559a1dfa6d73b881f0 2023-07-13T02:08:35.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 systemd[1]: Started Ceph mon.a for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:08:35.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-13T02:08:35.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: monmap e1: 1 mons at {a=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0]} removed_ranks: {} 2023-07-13T02:08:35.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: fsmap 2023-07-13T02:08:35.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: osdmap e1: 0 total, 0 up, 0 in 2023-07-13T02:08:35.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:35 smithi032 ceph-mon[104859]: mgrmap e1: no daemons active 2023-07-13T02:08:35.775 INFO:teuthology.orchestra.run.smithi032.stdout:Wrote config to /etc/ceph/ceph.conf 2023-07-13T02:08:35.775 INFO:teuthology.orchestra.run.smithi032.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-07-13T02:08:35.775 INFO:teuthology.orchestra.run.smithi032.stdout:Creating mgr... 2023-07-13T02:08:35.776 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying port 9283 ... 2023-07-13T02:08:35.776 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying port 8765 ... 2023-07-13T02:08:35.912 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mgr.y 2023-07-13T02:08:35.913 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to reset failed state of unit ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mgr.y.service: Unit ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mgr.y.service not loaded. 2023-07-13T02:08:36.010 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-11418e3c-2122-11ee-9b31-001a4aab830c.target.wants/ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mgr.y.service → /etc/systemd/system/ceph-11418e3c-2122-11ee-9b31-001a4aab830c@.service. 2023-07-13T02:08:36.017 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:36 smithi032 systemd[1]: Starting Ceph mgr.y for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:08:36.345 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-13T02:08:36.345 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2023-07-13T02:08:36.353 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-13T02:08:36.354 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2023-07-13T02:08:36.354 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2023-07-13T02:08:36.354 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to enable service . firewalld.service is not available 2023-07-13T02:08:36.362 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-13T02:08:36.362 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2023-07-13T02:08:36.372 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-13T02:08:36.372 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2023-07-13T02:08:36.372 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2023-07-13T02:08:36.372 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-07-13T02:08:36.373 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr to start... 2023-07-13T02:08:36.373 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr... 2023-07-13T02:08:36.635 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:36 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1467726919' entity='client.admin' 2023-07-13T02:08:36.636 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:36 smithi032 bash[105071]: 4952bc25fd9c1377ea32284b2dae86b161c9a2e701950f6989360e7c9e7e15d3 2023-07-13T02:08:36.636 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:36 smithi032 systemd[1]: Started Ceph mgr.y for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:08:36.636 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:36 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:36.468+0000 7fb3f687b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-13T02:08:37.010 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:36 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:36.659+0000 7fb3f687b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-13T02:08:37.175 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2023-07-13T02:08:37.176 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2023-07-13T02:08:37.176 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "11418e3c-2122-11ee-9b31-001a4aab830c", 2023-07-13T02:08:37.176 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2023-07-13T02:08:37.176 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-13T02:08:37.176 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-13T02:08:37.176 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-13T02:08:37.176 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:37.177 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-13T02:08:37.177 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-13T02:08:37.177 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2023-07-13T02:08:37.177 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2023-07-13T02:08:37.178 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-13T02:08:37.178 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "a" 2023-07-13T02:08:37.178 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2023-07-13T02:08:37.178 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 1, 2023-07-13T02:08:37.178 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-13T02:08:37.178 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-13T02:08:37.178 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-13T02:08:37.179 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-13T02:08:37.179 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:37.179 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-13T02:08:37.179 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-13T02:08:37.179 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-13T02:08:37.179 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-13T02:08:37.179 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-13T02:08:37.179 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-13T02:08:37.180 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-13T02:08:37.180 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-13T02:08:37.182 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:37.182 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-13T02:08:37.183 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-13T02:08:37.183 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-13T02:08:37.183 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-13T02:08:37.183 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-13T02:08:37.183 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-13T02:08:37.183 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-13T02:08:37.183 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-13T02:08:37.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-13T02:08:37.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:37.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-13T02:08:37.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-13T02:08:37.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-13T02:08:37.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-13T02:08:37.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:37.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-13T02:08:37.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-13T02:08:37.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-13T02:08:37.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-13T02:08:37.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-13T02:08:37.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-13T02:08:37.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2023-07-13T02:08:37.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2023-07-13T02:08:37.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-13T02:08:37.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:37.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-13T02:08:37.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-13T02:08:37.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2023-07-13T02:08:31.791604+0000", 2023-07-13T02:08:37.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-13T02:08:37.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:37.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-13T02:08:37.187 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2023-07-13T02:08:37.187 INFO:teuthology.orchestra.run.smithi032.stdout:mgr not available, waiting (1/15)... 2023-07-13T02:08:37.384 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:37 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:37.008+0000 7fb3f687b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-13T02:08:37.771 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:37 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/4067414552' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-13T02:08:37.772 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:37 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:37.536+0000 7fb3f687b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-13T02:08:37.772 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:37 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:37.769+0000 7fb3f687b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-13T02:08:38.635 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:38 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:38.201+0000 7fb3f687b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-13T02:08:39.288 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:38 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:38.937+0000 7fb3f687b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-13T02:08:39.289 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:39 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:39.027+0000 7fb3f687b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-13T02:08:39.862 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:39 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3832074614' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-13T02:08:39.862 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:39 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:39.679+0000 7fb3f687b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-13T02:08:39.863 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:39 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:39.768+0000 7fb3f687b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-13T02:08:39.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2023-07-13T02:08:39.926 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2023-07-13T02:08:39.926 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "11418e3c-2122-11ee-9b31-001a4aab830c", 2023-07-13T02:08:39.926 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2023-07-13T02:08:39.926 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-13T02:08:39.926 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-13T02:08:39.927 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-13T02:08:39.927 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:39.927 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-13T02:08:39.927 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-13T02:08:39.927 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2023-07-13T02:08:39.927 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2023-07-13T02:08:39.927 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-13T02:08:39.928 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "a" 2023-07-13T02:08:39.928 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2023-07-13T02:08:39.928 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2023-07-13T02:08:39.928 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-13T02:08:39.928 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-13T02:08:39.928 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-13T02:08:39.928 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-13T02:08:39.929 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:39.929 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-13T02:08:39.929 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-13T02:08:39.929 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-13T02:08:39.929 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-13T02:08:39.930 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-13T02:08:39.930 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-13T02:08:39.932 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-13T02:08:39.932 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-13T02:08:39.933 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:39.933 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-13T02:08:39.933 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-13T02:08:39.933 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-13T02:08:39.933 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-13T02:08:39.933 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-13T02:08:39.933 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-13T02:08:39.934 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-13T02:08:39.934 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-13T02:08:39.934 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-13T02:08:39.934 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:39.934 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-13T02:08:39.934 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-13T02:08:39.934 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-13T02:08:39.934 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-13T02:08:39.935 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:39.935 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-13T02:08:39.935 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-13T02:08:39.935 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-13T02:08:39.936 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-13T02:08:39.936 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-13T02:08:39.936 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-13T02:08:39.936 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2023-07-13T02:08:39.936 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2023-07-13T02:08:39.936 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-13T02:08:39.936 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:39.937 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-13T02:08:39.937 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-13T02:08:39.937 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2023-07-13T02:08:31.791604+0000", 2023-07-13T02:08:39.937 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-13T02:08:39.937 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:39.937 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-13T02:08:39.938 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2023-07-13T02:08:39.938 INFO:teuthology.orchestra.run.smithi032.stdout:mgr not available, waiting (2/15)... 2023-07-13T02:08:40.134 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:39 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:39.859+0000 7fb3f687b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-13T02:08:40.135 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:39 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:39.958+0000 7fb3f687b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-13T02:08:40.513 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:40 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:40.192+0000 7fb3f687b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-13T02:08:40.513 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:40 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:40.290+0000 7fb3f687b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-13T02:08:40.804 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:40 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:40.510+0000 7fb3f687b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-13T02:08:40.804 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:40 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:40.611+0000 7fb3f687b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-13T02:08:41.135 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:40 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:40.801+0000 7fb3f687b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-13T02:08:41.135 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:40 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:40.891+0000 7fb3f687b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-13T02:08:42.046 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:41 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:41.675+0000 7fb3f687b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-13T02:08:42.047 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:41 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:41.873+0000 7fb3f687b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-13T02:08:42.327 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:42 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:42.065+0000 7fb3f687b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-13T02:08:42.635 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:42 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3464646265' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-13T02:08:42.642 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2023-07-13T02:08:42.642 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2023-07-13T02:08:42.642 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "11418e3c-2122-11ee-9b31-001a4aab830c", 2023-07-13T02:08:42.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2023-07-13T02:08:42.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-13T02:08:42.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-13T02:08:42.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-13T02:08:42.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:42.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-13T02:08:42.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-13T02:08:42.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2023-07-13T02:08:42.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2023-07-13T02:08:42.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-13T02:08:42.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "a" 2023-07-13T02:08:42.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2023-07-13T02:08:42.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2023-07-13T02:08:42.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-13T02:08:42.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-13T02:08:42.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-13T02:08:42.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-13T02:08:42.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:42.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-13T02:08:42.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-13T02:08:42.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-13T02:08:42.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-13T02:08:42.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-13T02:08:42.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-13T02:08:42.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-13T02:08:42.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-13T02:08:42.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:42.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-13T02:08:42.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-13T02:08:42.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-13T02:08:42.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-13T02:08:42.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-13T02:08:42.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-13T02:08:42.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-13T02:08:42.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-13T02:08:42.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-13T02:08:42.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:42.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-13T02:08:42.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-13T02:08:42.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-13T02:08:42.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-13T02:08:42.652 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:42.652 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-13T02:08:42.652 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-13T02:08:42.652 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-13T02:08:42.652 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-13T02:08:42.652 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-13T02:08:42.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-13T02:08:42.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2023-07-13T02:08:42.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2023-07-13T02:08:42.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-13T02:08:42.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:42.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-13T02:08:42.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-13T02:08:42.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2023-07-13T02:08:31.791604+0000", 2023-07-13T02:08:42.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-13T02:08:42.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:42.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-13T02:08:42.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2023-07-13T02:08:42.655 INFO:teuthology.orchestra.run.smithi032.stdout:mgr not available, waiting (3/15)... 2023-07-13T02:08:43.015 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:42 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:42.703+0000 7fb3f687b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-13T02:08:43.016 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:42 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:42.890+0000 7fb3f687b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-13T02:08:43.351 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:43 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:43.080+0000 7fb3f687b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-13T02:08:43.351 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:43 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:43.199+0000 7fb3f687b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-13T02:08:43.635 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:43 smithi032 ceph-mon[104859]: Activating manager daemon y 2023-07-13T02:08:43.635 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:43 smithi032 ceph-mon[104859]: mgrmap e2: y(active, starting, since 0.00293762s) 2023-07-13T02:08:43.635 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:43 smithi032 ceph-mon[104859]: from='mgr.14100 172.21.15.32:0/1862309846' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-13T02:08:43.635 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:43 smithi032 ceph-mon[104859]: from='mgr.14100 172.21.15.32:0/1862309846' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-13T02:08:43.636 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:43 smithi032 ceph-mon[104859]: from='mgr.14100 172.21.15.32:0/1862309846' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-13T02:08:43.636 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:43 smithi032 ceph-mon[104859]: from='mgr.14100 172.21.15.32:0/1862309846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-13T02:08:43.636 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:43 smithi032 ceph-mon[104859]: from='mgr.14100 172.21.15.32:0/1862309846' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-13T02:08:43.636 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:43 smithi032 ceph-mon[104859]: Manager daemon y is now available 2023-07-13T02:08:43.637 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:43 smithi032 ceph-mon[104859]: from='mgr.14100 172.21.15.32:0/1862309846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-13T02:08:43.637 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:43 smithi032 ceph-mon[104859]: from='mgr.14100 172.21.15.32:0/1862309846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-13T02:08:43.637 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:43 smithi032 ceph-mon[104859]: from='mgr.14100 172.21.15.32:0/1862309846' entity='mgr.y' 2023-07-13T02:08:43.637 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:43 smithi032 ceph-mon[104859]: from='mgr.14100 172.21.15.32:0/1862309846' entity='mgr.y' 2023-07-13T02:08:43.637 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:43 smithi032 ceph-mon[104859]: from='mgr.14100 172.21.15.32:0/1862309846' entity='mgr.y' 2023-07-13T02:08:45.467 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:45 smithi032 ceph-mon[104859]: mgrmap e3: y(active, since 1.00705s) 2023-07-13T02:08:45.467 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:45 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3553713678' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-13T02:08:45.564 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2023-07-13T02:08:45.564 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2023-07-13T02:08:45.564 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "11418e3c-2122-11ee-9b31-001a4aab830c", 2023-07-13T02:08:45.565 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2023-07-13T02:08:45.565 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-13T02:08:45.565 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-13T02:08:45.565 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-13T02:08:45.566 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:45.566 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-13T02:08:45.566 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-13T02:08:45.566 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2023-07-13T02:08:45.567 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2023-07-13T02:08:45.567 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-13T02:08:45.567 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "a" 2023-07-13T02:08:45.567 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2023-07-13T02:08:45.568 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2023-07-13T02:08:45.568 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-13T02:08:45.568 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-13T02:08:45.569 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-13T02:08:45.569 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-13T02:08:45.569 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:45.569 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-13T02:08:45.570 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-13T02:08:45.570 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-13T02:08:45.570 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-13T02:08:45.570 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-13T02:08:45.571 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-13T02:08:45.571 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-13T02:08:45.571 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-13T02:08:45.572 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:45.572 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-13T02:08:45.572 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-13T02:08:45.572 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-13T02:08:45.573 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-13T02:08:45.573 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-13T02:08:45.573 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-13T02:08:45.573 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-13T02:08:45.574 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-13T02:08:45.574 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-13T02:08:45.574 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:45.574 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-13T02:08:45.575 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-13T02:08:45.575 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-13T02:08:45.575 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-13T02:08:45.575 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:45.576 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-13T02:08:45.576 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-13T02:08:45.576 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-13T02:08:45.577 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-13T02:08:45.577 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-13T02:08:45.577 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-13T02:08:45.577 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2023-07-13T02:08:45.578 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2023-07-13T02:08:45.578 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-13T02:08:45.578 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:45.578 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-13T02:08:45.579 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-13T02:08:45.579 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2023-07-13T02:08:31.791604+0000", 2023-07-13T02:08:45.579 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-13T02:08:45.580 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2023-07-13T02:08:45.580 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-13T02:08:45.580 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2023-07-13T02:08:45.581 INFO:teuthology.orchestra.run.smithi032.stdout:mgr is available 2023-07-13T02:08:46.709 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2023-07-13T02:08:46.709 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [global] 2023-07-13T02:08:46.710 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout fsid = 11418e3c-2122-11ee-9b31-001a4aab830c 2023-07-13T02:08:46.710 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-13T02:08:46.711 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-13T02:08:46.711 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-13T02:08:46.711 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-13T02:08:46.711 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2023-07-13T02:08:46.712 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-13T02:08:46.712 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-13T02:08:46.712 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2023-07-13T02:08:46.713 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [osd] 2023-07-13T02:08:46.713 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-13T02:08:46.713 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-13T02:08:46.714 INFO:teuthology.orchestra.run.smithi032.stdout:Enabling cephadm module... 2023-07-13T02:08:47.635 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:47 smithi032 ceph-mon[104859]: mgrmap e4: y(active, since 3s) 2023-07-13T02:08:47.635 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:47 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1237794827' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-13T02:08:47.635 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:47 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1237794827' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-07-13T02:08:48.528 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:48 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2520975079' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-07-13T02:08:48.529 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:48 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: ignoring --setuser ceph since I am not root 2023-07-13T02:08:48.529 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:48 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: ignoring --setgroup ceph since I am not root 2023-07-13T02:08:48.530 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:48 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:48.399+0000 7f02d78f4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-13T02:08:48.885 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:48 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:48.585+0000 7f02d78f4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-13T02:08:49.221 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:48 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:48.952+0000 7f02d78f4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-13T02:08:49.489 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2023-07-13T02:08:49.490 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-07-13T02:08:49.490 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-13T02:08:49.490 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "active_name": "y", 2023-07-13T02:08:49.490 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-13T02:08:49.490 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2023-07-13T02:08:49.490 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for the mgr to restart... 2023-07-13T02:08:49.490 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr epoch 5... 2023-07-13T02:08:49.596 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:49 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:49.469+0000 7f02d78f4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-13T02:08:49.596 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:49 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2520975079' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-07-13T02:08:49.597 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:49 smithi032 ceph-mon[104859]: mgrmap e5: y(active, since 5s) 2023-07-13T02:08:49.597 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:49 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2441822803' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-13T02:08:49.885 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:49 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:49.709+0000 7f02d78f4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-13T02:08:50.635 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:50 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:50.147+0000 7f02d78f4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-13T02:08:51.135 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:50 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:50.877+0000 7f02d78f4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-13T02:08:51.135 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:50 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:50.966+0000 7f02d78f4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-13T02:08:51.885 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:51 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:51.602+0000 7f02d78f4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-13T02:08:51.885 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:51 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:51.695+0000 7f02d78f4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-13T02:08:51.885 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:51 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:51.791+0000 7f02d78f4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-13T02:08:52.239 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:51 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:51.897+0000 7f02d78f4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-13T02:08:52.239 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:52 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:52.138+0000 7f02d78f4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-13T02:08:52.564 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:52 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:52.237+0000 7f02d78f4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-13T02:08:52.564 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:52 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:52.461+0000 7f02d78f4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-13T02:08:52.849 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:52 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:52.562+0000 7f02d78f4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-13T02:08:52.849 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:52 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:52.757+0000 7f02d78f4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-13T02:08:53.135 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:52 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:52.847+0000 7f02d78f4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-13T02:08:54.037 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:53 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:53.651+0000 7f02d78f4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-13T02:08:54.037 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:53 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:53.852+0000 7f02d78f4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-13T02:08:54.385 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:54 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:54.035+0000 7f02d78f4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-13T02:08:55.038 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:54 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:54.661+0000 7f02d78f4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-13T02:08:55.038 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:54 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:54.847+0000 7f02d78f4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-13T02:08:55.385 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:55 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:55.036+0000 7f02d78f4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-13T02:08:55.386 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:08:55 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:08:55.156+0000 7f02d78f4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-13T02:08:56.385 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:56 smithi032 ceph-mon[104859]: Active manager daemon y restarted 2023-07-13T02:08:56.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:56 smithi032 ceph-mon[104859]: Activating manager daemon y 2023-07-13T02:08:56.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:56 smithi032 ceph-mon[104859]: osdmap e2: 0 total, 0 up, 0 in 2023-07-13T02:08:56.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:56 smithi032 ceph-mon[104859]: mgrmap e6: y(active, starting, since 0.00325692s) 2023-07-13T02:08:56.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:56 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-13T02:08:56.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:56 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-13T02:08:56.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:56 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-13T02:08:56.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:56 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-13T02:08:56.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:56 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-13T02:08:56.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:56 smithi032 ceph-mon[104859]: Manager daemon y is now available 2023-07-13T02:08:56.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:56 smithi032 ceph-mon[104859]: Found migration_current of "None". Setting to last migration. 2023-07-13T02:08:56.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:56 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:08:56.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:56 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:08:56.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:56 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:08:56.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:56 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:08:56.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:56 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:08:56.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:56 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-13T02:08:56.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:56 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-13T02:08:56.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:56 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:08:56.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:56 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:08:57.635 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:57 smithi032 ceph-mon[104859]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-13T02:08:57.635 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:57 smithi032 ceph-mon[104859]: mgrmap e7: y(active, since 1.00535s) 2023-07-13T02:08:57.636 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:57 smithi032 ceph-mon[104859]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-13T02:08:57.636 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:57 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:08:58.431 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2023-07-13T02:08:58.432 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-07-13T02:08:58.432 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-13T02:08:58.432 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2023-07-13T02:08:58.432 INFO:teuthology.orchestra.run.smithi032.stdout:mgr epoch 5 is available 2023-07-13T02:08:58.433 INFO:teuthology.orchestra.run.smithi032.stdout:Setting orchestrator backend to cephadm... 2023-07-13T02:08:58.434 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:58 smithi032 ceph-mon[104859]: [13/Jul/2023:02:08:56] ENGINE Bus STARTING 2023-07-13T02:08:58.434 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:58 smithi032 ceph-mon[104859]: [13/Jul/2023:02:08:56] ENGINE Serving on http://172.21.15.32:8765 2023-07-13T02:08:58.435 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:58 smithi032 ceph-mon[104859]: [13/Jul/2023:02:08:56] ENGINE Serving on https://172.21.15.32:7150 2023-07-13T02:08:58.435 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:58 smithi032 ceph-mon[104859]: [13/Jul/2023:02:08:56] ENGINE Bus STARTED 2023-07-13T02:08:59.336 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:59 smithi032 ceph-mon[104859]: mgrmap e8: y(active, since 3s) 2023-07-13T02:08:59.337 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:59 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:08:59.337 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:08:59 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:00.055 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout value unchanged 2023-07-13T02:09:00.055 INFO:teuthology.orchestra.run.smithi032.stdout:Generating ssh key... 2023-07-13T02:09:00.635 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:00 smithi032 ceph-mon[104859]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:01.340 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:01 smithi032 ceph-mon[104859]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:01.341 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:01 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:09:01.341 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:01 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:09:01.341 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:01 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: Generating public/private rsa key pair. 2023-07-13T02:09:01.341 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:01 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: Your identification has been saved in /tmp/tmpn7sini0l/key. 2023-07-13T02:09:01.341 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:01 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: Your public key has been saved in /tmp/tmpn7sini0l/key.pub. 2023-07-13T02:09:01.342 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:01 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: The key fingerprint is: 2023-07-13T02:09:01.342 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:01 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: SHA256:1cByP5ingT+sNKu25U/f5GZUKCZ1fXGNU7gb7L13srI ceph-11418e3c-2122-11ee-9b31-001a4aab830c 2023-07-13T02:09:01.342 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:01 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: The key's randomart image is: 2023-07-13T02:09:01.342 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:01 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: +---[RSA 3072]----+ 2023-07-13T02:09:01.342 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:01 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: | .. *=| 2023-07-13T02:09:01.342 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:01 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: | . oo. = =| 2023-07-13T02:09:01.342 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:01 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: | +.=.o +.| 2023-07-13T02:09:01.342 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:01 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: | ..= * = .| 2023-07-13T02:09:01.343 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:01 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: | So * + = | 2023-07-13T02:09:01.343 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:01 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: | o = + .| 2023-07-13T02:09:01.343 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:01 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: | ..+.. .. .| 2023-07-13T02:09:01.343 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:01 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: | .oo. ..++.o| 2023-07-13T02:09:01.343 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:01 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: | .oo... E=+oo| 2023-07-13T02:09:01.343 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:01 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: +----[SHA256]-----+ 2023-07-13T02:09:02.215 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:02 smithi032 ceph-mon[104859]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:02.216 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:02 smithi032 ceph-mon[104859]: Generating ssh key... 2023-07-13T02:09:02.255 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDa5+PH8/0p1ZmqUFDEW0FXYMwpXecPY7ohJWHV4G2MfE4f4Cjzp1D5DVtj70VNY1VnGcSrfnxwosxBbJ1Hft6kL1sKwkoKZrr04C1svQ91o43GulBnaVegQhAj2QATw8MQpwMFbUhIS5K73TcQibCCH7r5LH86rW5Ms6060GlcwqGYHePDSyqm3Q80HcdmfD0YkqAKJerYOkls/er7QWmbzBZTtqdhZbuIpGAaRW+T50XXvpSBFP/vsUDDNUxwhnPIsxLM9w0Fc+57IF+59WgtKHUl+vBsMAE0M0tm6/p78Ky3jveJqZ/B0LvzgD7BS4rdJp1tI2j2jVHI7IvhF/8iGwO849d4smxPnefbx/unanKXd8/1TvMDXivixFw9zLkNQZnnCrVjGGPkLmKMRXJ0D7ffXIMn7UagzDu5zKgea5MSBD7Lc5JjZXDDqtUrOMxB0tmcc5/yQwJaH8l3hA12cFtDmUTZkC9pU+2MgIHrlzo5unY8593az5W2g31N1+U= ceph-11418e3c-2122-11ee-9b31-001a4aab830c 2023-07-13T02:09:02.256 INFO:teuthology.orchestra.run.smithi032.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-07-13T02:09:02.256 INFO:teuthology.orchestra.run.smithi032.stdout:Adding key to root@localhost authorized_keys... 2023-07-13T02:09:02.256 INFO:teuthology.orchestra.run.smithi032.stdout:Adding host smithi032... 2023-07-13T02:09:03.288 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:03 smithi032 ceph-mon[104859]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:04.481 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:04 smithi032 ceph-mon[104859]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi032", "addr": "172.21.15.32", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:04.481 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:04 smithi032 ceph-mon[104859]: Deploying cephadm binary to smithi032 2023-07-13T02:09:05.238 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Added host 'smithi032' with addr '172.21.15.32' 2023-07-13T02:09:05.238 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying unmanaged mon service... 2023-07-13T02:09:06.104 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:05 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:09:06.104 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:05 smithi032 ceph-mon[104859]: Added host smithi032 2023-07-13T02:09:06.104 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:05 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:06.105 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:05 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:09:06.237 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-07-13T02:09:06.238 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying unmanaged mgr service... 2023-07-13T02:09:07.281 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:06 smithi032 ceph-mon[104859]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:07.282 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:06 smithi032 ceph-mon[104859]: Saving service mon spec with placement count:5 2023-07-13T02:09:07.282 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:06 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:09:07.291 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-07-13T02:09:08.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:07 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:09:08.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:07 smithi032 ceph-mon[104859]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:08.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:07 smithi032 ceph-mon[104859]: Saving service mgr spec with placement count:2 2023-07-13T02:09:08.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:07 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:09:08.107 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:07 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2150654459' entity='client.admin' 2023-07-13T02:09:09.056 INFO:teuthology.orchestra.run.smithi032.stdout:Enabling the dashboard module... 2023-07-13T02:09:09.385 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:09 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:09:09.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:09 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3403984800' entity='client.admin' 2023-07-13T02:09:09.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:09 smithi032 ceph-mon[104859]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:09:10.315 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:10 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/13468188' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-07-13T02:09:10.316 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:10 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: ignoring --setuser ceph since I am not root 2023-07-13T02:09:10.316 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:10 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: ignoring --setgroup ceph since I am not root 2023-07-13T02:09:10.316 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:10 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:10.223+0000 7f7de5ada200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-13T02:09:10.637 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:10 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:10.409+0000 7f7de5ada200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-13T02:09:11.045 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:10 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:10.761+0000 7f7de5ada200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-13T02:09:11.271 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2023-07-13T02:09:11.271 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-07-13T02:09:11.272 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-13T02:09:11.272 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "active_name": "y", 2023-07-13T02:09:11.272 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-13T02:09:11.272 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2023-07-13T02:09:11.273 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for the mgr to restart... 2023-07-13T02:09:11.273 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr epoch 9... 2023-07-13T02:09:11.377 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:11 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:11.286+0000 7f7de5ada200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-13T02:09:11.378 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:11 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/13468188' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-07-13T02:09:11.378 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:11 smithi032 ceph-mon[104859]: mgrmap e9: y(active, since 14s) 2023-07-13T02:09:11.378 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:11 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/982001257' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-13T02:09:11.635 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:11 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:11.525+0000 7f7de5ada200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-13T02:09:12.385 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:11 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:11.962+0000 7f7de5ada200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-13T02:09:13.135 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:12 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:12.698+0000 7f7de5ada200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-13T02:09:13.136 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:12 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:12.790+0000 7f7de5ada200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-13T02:09:13.708 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:13 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:13.427+0000 7f7de5ada200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-13T02:09:13.709 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:13 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:13.515+0000 7f7de5ada200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-13T02:09:13.709 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:13 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:13.608+0000 7f7de5ada200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-13T02:09:14.042 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:13 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:13.705+0000 7f7de5ada200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-13T02:09:14.042 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:13 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:13.941+0000 7f7de5ada200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-13T02:09:14.364 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:14 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:14.039+0000 7f7de5ada200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-13T02:09:14.365 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:14 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:14.261+0000 7f7de5ada200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-13T02:09:14.635 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:14 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:14.362+0000 7f7de5ada200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-13T02:09:14.636 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:14 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:14.554+0000 7f7de5ada200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-13T02:09:15.136 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:14 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:14.644+0000 7f7de5ada200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-13T02:09:15.820 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:15 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:15.432+0000 7f7de5ada200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-13T02:09:15.820 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:15 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:15.633+0000 7f7de5ada200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-13T02:09:16.830 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:15 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:15.817+0000 7f7de5ada200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-13T02:09:16.830 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:16 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:16.449+0000 7f7de5ada200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-13T02:09:16.830 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:16 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:16.637+0000 7f7de5ada200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-13T02:09:16.830 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:16 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:16.827+0000 7f7de5ada200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-13T02:09:17.084 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:09:16 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:09:16.947+0000 7f7de5ada200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-13T02:09:17.085 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:17 smithi032 ceph-mon[104859]: Active manager daemon y restarted 2023-07-13T02:09:17.085 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:17 smithi032 ceph-mon[104859]: Activating manager daemon y 2023-07-13T02:09:17.085 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:17 smithi032 ceph-mon[104859]: osdmap e3: 0 total, 0 up, 0 in 2023-07-13T02:09:17.085 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:17 smithi032 ceph-mon[104859]: mgrmap e10: y(active, starting, since 0.00347432s) 2023-07-13T02:09:17.085 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:17 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-13T02:09:17.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:17 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-13T02:09:17.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:17 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-13T02:09:17.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:17 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-13T02:09:17.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:17 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-13T02:09:17.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:17 smithi032 ceph-mon[104859]: Manager daemon y is now available 2023-07-13T02:09:17.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:17 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:17.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:17 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:18.329 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2023-07-13T02:09:18.329 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-07-13T02:09:18.329 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-13T02:09:18.330 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2023-07-13T02:09:18.330 INFO:teuthology.orchestra.run.smithi032.stdout:mgr epoch 9 is available 2023-07-13T02:09:18.330 INFO:teuthology.orchestra.run.smithi032.stdout:Generating a dashboard self-signed certificate... 2023-07-13T02:09:18.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:18 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-13T02:09:18.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:18 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-13T02:09:18.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:18 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:18.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:18 smithi032 ceph-mon[104859]: mgrmap e11: y(active, since 1.0056s) 2023-07-13T02:09:19.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:19 smithi032 ceph-mon[104859]: [13/Jul/2023:02:09:18] ENGINE Bus STARTING 2023-07-13T02:09:19.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:19 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:19.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:19 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:19.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:19 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:20.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:20 smithi032 ceph-mon[104859]: [13/Jul/2023:02:09:18] ENGINE Serving on http://172.21.15.32:8765 2023-07-13T02:09:20.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:20 smithi032 ceph-mon[104859]: [13/Jul/2023:02:09:18] ENGINE Serving on https://172.21.15.32:7150 2023-07-13T02:09:20.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:20 smithi032 ceph-mon[104859]: [13/Jul/2023:02:09:18] ENGINE Bus STARTED 2023-07-13T02:09:20.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:20 smithi032 ceph-mon[104859]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:20.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:20 smithi032 ceph-mon[104859]: mgrmap e12: y(active, since 2s) 2023-07-13T02:09:20.403 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-07-13T02:09:20.403 INFO:teuthology.orchestra.run.smithi032.stdout:Creating initial admin user... 2023-07-13T02:09:21.438 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:21 smithi032 ceph-mon[104859]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:21.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$AHUKT9kuDem67Sr12A/1U.LyokM3vyhRwxgJ04KHO6bGq5PnNAecG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1689214161, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-07-13T02:09:21.504 INFO:teuthology.orchestra.run.smithi032.stdout:Fetching dashboard port number... 2023-07-13T02:09:22.297 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:22 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:22.297 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:22 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/654036068' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-07-13T02:09:22.320 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 8443 2023-07-13T02:09:22.328 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-13T02:09:22.329 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2023-07-13T02:09:22.337 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-13T02:09:22.337 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2023-07-13T02:09:22.337 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2023-07-13T02:09:22.338 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-07-13T02:09:22.339 INFO:teuthology.orchestra.run.smithi032.stdout:Ceph Dashboard is now available at: 2023-07-13T02:09:22.339 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:09:22.339 INFO:teuthology.orchestra.run.smithi032.stdout: URL: https://smithi032.front.sepia.ceph.com:8443/ 2023-07-13T02:09:22.340 INFO:teuthology.orchestra.run.smithi032.stdout: User: admin 2023-07-13T02:09:22.340 INFO:teuthology.orchestra.run.smithi032.stdout: Password: v30ge85w10 2023-07-13T02:09:22.340 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:09:22.340 INFO:teuthology.orchestra.run.smithi032.stdout:Saving cluster configuration to /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/config directory 2023-07-13T02:09:22.341 INFO:teuthology.orchestra.run.smithi032.stdout:Enabling autotune for osd_memory_target 2023-07-13T02:09:24.061 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-07-13T02:09:24.062 INFO:teuthology.orchestra.run.smithi032.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-07-13T02:09:24.062 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:09:24.062 INFO:teuthology.orchestra.run.smithi032.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-07-13T02:09:24.062 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:09:24.062 INFO:teuthology.orchestra.run.smithi032.stdout:Or, if you are only running a single cluster on this host: 2023-07-13T02:09:24.062 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:09:24.063 INFO:teuthology.orchestra.run.smithi032.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-07-13T02:09:24.063 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:09:24.063 INFO:teuthology.orchestra.run.smithi032.stdout:Please consider enabling telemetry to help improve Ceph: 2023-07-13T02:09:24.063 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:09:24.063 INFO:teuthology.orchestra.run.smithi032.stdout: ceph telemetry on 2023-07-13T02:09:24.063 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:09:24.063 INFO:teuthology.orchestra.run.smithi032.stdout:For more information see: 2023-07-13T02:09:24.064 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:09:24.064 INFO:teuthology.orchestra.run.smithi032.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-07-13T02:09:24.064 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:09:24.064 INFO:teuthology.orchestra.run.smithi032.stdout:Bootstrap complete. 2023-07-13T02:09:24.097 INFO:tasks.cephadm:Fetching config... 2023-07-13T02:09:24.097 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2023-07-13T02:09:24.098 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-07-13T02:09:24.116 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-07-13T02:09:24.117 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2023-07-13T02:09:24.117 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-07-13T02:09:24.177 INFO:tasks.cephadm:Fetching mon keyring... 2023-07-13T02:09:24.178 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2023-07-13T02:09:24.178 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/keyring of=/dev/stdout 2023-07-13T02:09:24.244 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:24 smithi032 ceph-mon[104859]: mgrmap e13: y(active, since 6s) 2023-07-13T02:09:24.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:24 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1608105501' entity='client.admin' 2023-07-13T02:09:24.248 INFO:tasks.cephadm:Fetching pub ssh key... 2023-07-13T02:09:24.249 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2023-07-13T02:09:24.249 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-07-13T02:09:24.306 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-07-13T02:09:24.307 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDa5+PH8/0p1ZmqUFDEW0FXYMwpXecPY7ohJWHV4G2MfE4f4Cjzp1D5DVtj70VNY1VnGcSrfnxwosxBbJ1Hft6kL1sKwkoKZrr04C1svQ91o43GulBnaVegQhAj2QATw8MQpwMFbUhIS5K73TcQibCCH7r5LH86rW5Ms6060GlcwqGYHePDSyqm3Q80HcdmfD0YkqAKJerYOkls/er7QWmbzBZTtqdhZbuIpGAaRW+T50XXvpSBFP/vsUDDNUxwhnPIsxLM9w0Fc+57IF+59WgtKHUl+vBsMAE0M0tm6/p78Ky3jveJqZ/B0LvzgD7BS4rdJp1tI2j2jVHI7IvhF/8iGwO849d4smxPnefbx/unanKXd8/1TvMDXivixFw9zLkNQZnnCrVjGGPkLmKMRXJ0D7ffXIMn7UagzDu5zKgea5MSBD7Lc5JjZXDDqtUrOMxB0tmcc5/yQwJaH8l3hA12cFtDmUTZkC9pU+2MgIHrlzo5unY8593az5W2g31N1+U= ceph-11418e3c-2122-11ee-9b31-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-13T02:09:24.385 INFO:teuthology.orchestra.run.smithi032.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDa5+PH8/0p1ZmqUFDEW0FXYMwpXecPY7ohJWHV4G2MfE4f4Cjzp1D5DVtj70VNY1VnGcSrfnxwosxBbJ1Hft6kL1sKwkoKZrr04C1svQ91o43GulBnaVegQhAj2QATw8MQpwMFbUhIS5K73TcQibCCH7r5LH86rW5Ms6060GlcwqGYHePDSyqm3Q80HcdmfD0YkqAKJerYOkls/er7QWmbzBZTtqdhZbuIpGAaRW+T50XXvpSBFP/vsUDDNUxwhnPIsxLM9w0Fc+57IF+59WgtKHUl+vBsMAE0M0tm6/p78Ky3jveJqZ/B0LvzgD7BS4rdJp1tI2j2jVHI7IvhF/8iGwO849d4smxPnefbx/unanKXd8/1TvMDXivixFw9zLkNQZnnCrVjGGPkLmKMRXJ0D7ffXIMn7UagzDu5zKgea5MSBD7Lc5JjZXDDqtUrOMxB0tmcc5/yQwJaH8l3hA12cFtDmUTZkC9pU+2MgIHrlzo5unY8593az5W2g31N1+U= ceph-11418e3c-2122-11ee-9b31-001a4aab830c 2023-07-13T02:09:24.397 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDa5+PH8/0p1ZmqUFDEW0FXYMwpXecPY7ohJWHV4G2MfE4f4Cjzp1D5DVtj70VNY1VnGcSrfnxwosxBbJ1Hft6kL1sKwkoKZrr04C1svQ91o43GulBnaVegQhAj2QATw8MQpwMFbUhIS5K73TcQibCCH7r5LH86rW5Ms6060GlcwqGYHePDSyqm3Q80HcdmfD0YkqAKJerYOkls/er7QWmbzBZTtqdhZbuIpGAaRW+T50XXvpSBFP/vsUDDNUxwhnPIsxLM9w0Fc+57IF+59WgtKHUl+vBsMAE0M0tm6/p78Ky3jveJqZ/B0LvzgD7BS4rdJp1tI2j2jVHI7IvhF/8iGwO849d4smxPnefbx/unanKXd8/1TvMDXivixFw9zLkNQZnnCrVjGGPkLmKMRXJ0D7ffXIMn7UagzDu5zKgea5MSBD7Lc5JjZXDDqtUrOMxB0tmcc5/yQwJaH8l3hA12cFtDmUTZkC9pU+2MgIHrlzo5unY8593az5W2g31N1+U= ceph-11418e3c-2122-11ee-9b31-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-13T02:09:24.443 INFO:teuthology.orchestra.run.smithi191.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDa5+PH8/0p1ZmqUFDEW0FXYMwpXecPY7ohJWHV4G2MfE4f4Cjzp1D5DVtj70VNY1VnGcSrfnxwosxBbJ1Hft6kL1sKwkoKZrr04C1svQ91o43GulBnaVegQhAj2QATw8MQpwMFbUhIS5K73TcQibCCH7r5LH86rW5Ms6060GlcwqGYHePDSyqm3Q80HcdmfD0YkqAKJerYOkls/er7QWmbzBZTtqdhZbuIpGAaRW+T50XXvpSBFP/vsUDDNUxwhnPIsxLM9w0Fc+57IF+59WgtKHUl+vBsMAE0M0tm6/p78Ky3jveJqZ/B0LvzgD7BS4rdJp1tI2j2jVHI7IvhF/8iGwO849d4smxPnefbx/unanKXd8/1TvMDXivixFw9zLkNQZnnCrVjGGPkLmKMRXJ0D7ffXIMn7UagzDu5zKgea5MSBD7Lc5JjZXDDqtUrOMxB0tmcc5/yQwJaH8l3hA12cFtDmUTZkC9pU+2MgIHrlzo5unY8593az5W2g31N1+U= ceph-11418e3c-2122-11ee-9b31-001a4aab830c 2023-07-13T02:09:24.456 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-07-13T02:09:24.692 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:09:26.373 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-07-13T02:09:26.374 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-07-13T02:09:26.635 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:09:27.034 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:26 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/890431937' entity='client.admin' 2023-07-13T02:09:28.176 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi191 2023-07-13T02:09:28.176 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-07-13T02:09:28.177 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/etc/ceph/ceph.conf 2023-07-13T02:09:28.199 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-07-13T02:09:28.199 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-13T02:09:28.260 INFO:tasks.cephadm:Adding host smithi191 to orchestrator... 2023-07-13T02:09:28.260 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph orch host add smithi191 2023-07-13T02:09:28.497 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:09:28.636 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:28 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:28.636 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:28 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:28.637 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:28 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi032", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:09:28.637 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:28 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:28.637 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:28 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:28.637 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:28 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:28.637 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:28 smithi032 ceph-mon[104859]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:28.637 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:28 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:28.637 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:28 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:28.638 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:28 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:28.638 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:28 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:09:28.638 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:28 smithi032 ceph-mon[104859]: Updating smithi032:/etc/ceph/ceph.conf 2023-07-13T02:09:30.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:29 smithi032 ceph-mon[104859]: Updating smithi032:/var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-13T02:09:30.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:29 smithi032 ceph-mon[104859]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2023-07-13T02:09:30.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:29 smithi032 ceph-mon[104859]: Updating smithi032:/var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/config/ceph.client.admin.keyring 2023-07-13T02:09:30.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:29 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:30.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:29 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:30.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:29 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:31.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:31 smithi032 ceph-mon[104859]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi191", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:31.812 INFO:teuthology.orchestra.run.smithi032.stdout:Added host 'smithi191' with addr '172.21.15.191' 2023-07-13T02:09:32.122 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:32 smithi032 ceph-mon[104859]: Deploying cephadm binary to smithi191 2023-07-13T02:09:32.123 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:32 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:32.123 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:32 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:32.173 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph orch host ls --format=json 2023-07-13T02:09:32.405 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:09:33.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:33 smithi032 ceph-mon[104859]: Added host smithi191 2023-07-13T02:09:33.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:33 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:33.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:33 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:33.465 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:09:33.466 INFO:teuthology.orchestra.run.smithi032.stdout:[{"addr": "172.21.15.32", "hostname": "smithi032", "labels": [], "status": ""}, {"addr": "172.21.15.191", "hostname": "smithi191", "labels": [], "status": ""}] 2023-07-13T02:09:33.864 INFO:tasks.cephadm:Setting crush tunables to default 2023-07-13T02:09:33.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd crush tunables default 2023-07-13T02:09:34.093 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:09:34.537 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:34 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:34.538 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:34 smithi032 ceph-mon[104859]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-13T02:09:36.057 INFO:teuthology.orchestra.run.smithi032.stderr:adjusted tunables profile to default 2023-07-13T02:09:36.354 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:36 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3984784913' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-13T02:09:36.354 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:36 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:36.355 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:36 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:36.355 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:36 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:36.355 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:36 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:36.355 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:36 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:09:36.355 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:36 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:36.356 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:36 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:09:36.438 INFO:tasks.cephadm:Adding mon.a on smithi032 2023-07-13T02:09:36.439 INFO:tasks.cephadm:Adding mon.c on smithi032 2023-07-13T02:09:36.439 INFO:tasks.cephadm:Adding mon.b on smithi191 2023-07-13T02:09:36.440 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph orch apply mon '3;smithi032:172.21.15.32=a;smithi032:[v2:172.21.15.32:3301,v1:172.21.15.32:6790]=c;smithi191:172.21.15.191=b' 2023-07-13T02:09:36.688 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-13T02:09:37.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:37 smithi032 ceph-mon[104859]: Updating smithi191:/etc/ceph/ceph.conf 2023-07-13T02:09:37.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:37 smithi032 ceph-mon[104859]: Updating smithi191:/var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-13T02:09:37.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:37 smithi032 ceph-mon[104859]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2023-07-13T02:09:37.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:37 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3984784913' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-13T02:09:37.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:37 smithi032 ceph-mon[104859]: osdmap e4: 0 total, 0 up, 0 in 2023-07-13T02:09:37.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:37 smithi032 ceph-mon[104859]: Updating smithi191:/var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/config/ceph.client.admin.keyring 2023-07-13T02:09:37.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:37 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:37.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:37 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:37.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:37 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:37.730 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mon update... 2023-07-13T02:09:38.140 DEBUG:teuthology.orchestra.run.smithi032:mon.c> sudo journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mon.c.service 2023-07-13T02:09:38.143 DEBUG:teuthology.orchestra.run.smithi191:mon.b> sudo journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mon.b.service 2023-07-13T02:09:38.146 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-13T02:09:38.146 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph mon dump -f json 2023-07-13T02:09:38.176 INFO:journalctl@ceph.mon.b.smithi191.stdout:-- Logs begin at Thu 2023-07-13 01:53:22 UTC. -- 2023-07-13T02:09:38.181 INFO:journalctl@ceph.mon.c.smithi032.stdout:-- Logs begin at Thu 2023-07-13 01:52:05 UTC. -- 2023-07-13T02:09:38.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:38 smithi032 ceph-mon[104859]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:38.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:38 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:38.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:38 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:38.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:38 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:38.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:38 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:09:38.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:38 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:38.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:38 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-13T02:09:38.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:38 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:38.652 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-13T02:09:39.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:39 smithi032 ceph-mon[104859]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi032:172.21.15.32=a;smithi032:[v2:172.21.15.32:3301,v1:172.21.15.32:6790]=c;smithi191:172.21.15.191=b", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:39.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:39 smithi032 ceph-mon[104859]: Saving service mon spec with placement smithi032:172.21.15.32=a;smithi032:[v2:172.21.15.32:3301,v1:172.21.15.32:6790]=c;smithi191:172.21.15.191=b;count:3 2023-07-13T02:09:39.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:39 smithi032 ceph-mon[104859]: Deploying daemon mon.b on smithi191 2023-07-13T02:09:40.301 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-13T02:09:40.301 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":1,"fsid":"11418e3c-2122-11ee-9b31-001a4aab830c","modified":"2023-07-13T02:08:30.011732Z","created":"2023-07-13T02:08:30.011732Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-07-13T02:09:40.303 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 1 2023-07-13T02:09:40.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:40 smithi032 ceph-mon[104859]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:40.710 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:40 smithi191 ceph-mon[108588]: expand_channel_meta expanded map: {default=debug} 2023-07-13T02:09:41.712 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-13T02:09:41.712 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph mon dump -f json 2023-07-13T02:09:41.948 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.b/config 2023-07-13T02:09:45.886 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:45 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-13T02:09:45.887 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:45 smithi032 ceph-mon[104859]: mon.a calling monitor election 2023-07-13T02:09:45.887 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:45 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:45.887 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:45 smithi032 ceph-mon[104859]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:45.887 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:45 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:45.887 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:45 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:45.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:45 smithi032 ceph-mon[104859]: mon.b calling monitor election 2023-07-13T02:09:45.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:45 smithi032 ceph-mon[104859]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:45.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:45 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:45.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:45 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:45.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:45 smithi032 ceph-mon[104859]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:45.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:45 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:45.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:45 smithi032 ceph-mon[104859]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2023-07-13T02:09:45.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:45 smithi032 ceph-mon[104859]: monmap e2: 2 mons at {a=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0],b=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0]} removed_ranks: {} 2023-07-13T02:09:45.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:45 smithi032 ceph-mon[104859]: fsmap 2023-07-13T02:09:45.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:45 smithi032 ceph-mon[104859]: osdmap e4: 0 total, 0 up, 0 in 2023-07-13T02:09:45.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:45 smithi032 ceph-mon[104859]: mgrmap e13: y(active, since 28s) 2023-07-13T02:09:45.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:45 smithi032 ceph-mon[104859]: overall HEALTH_OK 2023-07-13T02:09:45.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:45 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:45.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:45 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:45.980 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:45 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-13T02:09:45.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:45 smithi191 ceph-mon[108588]: mon.a calling monitor election 2023-07-13T02:09:45.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:45 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:45.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:45 smithi191 ceph-mon[108588]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:45.982 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:45 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:45.982 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:45 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:45.982 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:45 smithi191 ceph-mon[108588]: mon.b calling monitor election 2023-07-13T02:09:45.982 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:45 smithi191 ceph-mon[108588]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:45.982 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:45 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:45.982 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:45 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:45.982 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:45 smithi191 ceph-mon[108588]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:45.983 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:45 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:45.983 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:45 smithi191 ceph-mon[108588]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2023-07-13T02:09:45.983 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:45 smithi191 ceph-mon[108588]: monmap e2: 2 mons at {a=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0],b=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0]} removed_ranks: {} 2023-07-13T02:09:45.983 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:45 smithi191 ceph-mon[108588]: fsmap 2023-07-13T02:09:45.984 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:45 smithi191 ceph-mon[108588]: osdmap e4: 0 total, 0 up, 0 in 2023-07-13T02:09:45.984 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:45 smithi191 ceph-mon[108588]: mgrmap e13: y(active, since 28s) 2023-07-13T02:09:45.984 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:45 smithi191 ceph-mon[108588]: overall HEALTH_OK 2023-07-13T02:09:45.985 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:45 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:45.985 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:45 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:46.144 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-13T02:09:46.145 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":2,"fsid":"11418e3c-2122-11ee-9b31-001a4aab830c","modified":"2023-07-13T02:09:40.387459Z","created":"2023-07-13T02:08:30.011732Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-13T02:09:46.147 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 2 2023-07-13T02:09:46.574 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:46 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:46.574 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:46 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-13T02:09:46.574 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:46 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:46.574 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:46 smithi032 ceph-mon[104859]: Deploying daemon mon.c on smithi032 2023-07-13T02:09:46.574 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:46 smithi032 ceph-mon[104859]: from='client.? 172.21.15.191:0/930590995' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-13T02:09:46.575 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:46 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:46.729 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:46 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:46.730 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:46 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-13T02:09:46.730 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:46 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:46.731 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:46 smithi191 ceph-mon[108588]: Deploying daemon mon.c on smithi032 2023-07-13T02:09:46.731 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:46 smithi191 ceph-mon[108588]: from='client.? 172.21.15.191:0/930590995' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-13T02:09:46.731 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:46 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:47.385 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 systemd[1]: Starting Ceph mon.c for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:09:47.469 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-13T02:09:47.470 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph mon dump -f json 2023-07-13T02:09:47.703 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.b/config 2023-07-13T02:09:47.887 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: set uid:gid to 167:167 (ceph:ceph) 2023-07-13T02:09:47.887 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: ceph version 18.0.0-4093-g3990f91a (3990f91ac125621431ef1b77c2fc96421bec9e51) reef (dev), process ceph-mon, pid 2 2023-07-13T02:09:47.887 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: pidfile_write: ignore empty --pid-file 2023-07-13T02:09:47.888 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 bash[116791]: f05e4292e65755ab236909c57156814336dd49ed5dc402bb23d97d4cefda2a34 2023-07-13T02:09:47.888 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 systemd[1]: Started Ceph mon.c for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:09:47.888 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: load: jerasure load: lrc 2023-07-13T02:09:47.888 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: RocksDB version: 7.9.2 2023-07-13T02:09:47.888 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Git sha 0 2023-07-13T02:09:47.888 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Compile date 2023-07-12 14:33:48 2023-07-13T02:09:47.888 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: DB SUMMARY 2023-07-13T02:09:47.889 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: DB Session ID: Q0OA1WF7JR16NS9Q0B7T 2023-07-13T02:09:47.889 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: CURRENT file: CURRENT 2023-07-13T02:09:47.889 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: IDENTITY file: IDENTITY 2023-07-13T02:09:47.890 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-13T02:09:47.890 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-07-13T02:09:47.890 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2023-07-13T02:09:47.890 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.error_if_exists: 0 2023-07-13T02:09:47.891 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.create_if_missing: 0 2023-07-13T02:09:47.891 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.paranoid_checks: 1 2023-07-13T02:09:47.891 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-13T02:09:47.891 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-13T02:09:47.892 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-13T02:09:47.892 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.env: 0x55a52d68d720 2023-07-13T02:09:47.893 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.fs: PosixFileSystem 2023-07-13T02:09:47.893 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.info_log: 0x55a52f5ef3c0 2023-07-13T02:09:47.893 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_file_opening_threads: 16 2023-07-13T02:09:47.893 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.statistics: (nil) 2023-07-13T02:09:47.894 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.use_fsync: 0 2023-07-13T02:09:47.894 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_log_file_size: 0 2023-07-13T02:09:47.894 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-13T02:09:47.895 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-13T02:09:47.895 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.keep_log_file_num: 1000 2023-07-13T02:09:47.895 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.recycle_log_file_num: 0 2023-07-13T02:09:47.896 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.allow_fallocate: 1 2023-07-13T02:09:47.896 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.allow_mmap_reads: 0 2023-07-13T02:09:47.896 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.allow_mmap_writes: 0 2023-07-13T02:09:47.897 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.use_direct_reads: 0 2023-07-13T02:09:47.897 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-13T02:09:47.897 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.create_missing_column_families: 0 2023-07-13T02:09:47.897 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.db_log_dir: 2023-07-13T02:09:47.900 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.wal_dir: 2023-07-13T02:09:47.900 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-13T02:09:47.901 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-13T02:09:47.901 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-13T02:09:47.901 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-13T02:09:47.901 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-13T02:09:47.901 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-13T02:09:47.901 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.advise_random_on_open: 1 2023-07-13T02:09:47.901 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.db_write_buffer_size: 0 2023-07-13T02:09:47.902 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.write_buffer_manager: 0x55a52e8603c0 2023-07-13T02:09:47.902 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-13T02:09:47.902 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-13T02:09:47.902 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-13T02:09:47.902 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.rate_limiter: (nil) 2023-07-13T02:09:47.902 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-13T02:09:47.902 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.wal_recovery_mode: 2 2023-07-13T02:09:47.903 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.enable_thread_tracking: 0 2023-07-13T02:09:47.903 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.enable_pipelined_write: 0 2023-07-13T02:09:47.903 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.unordered_write: 0 2023-07-13T02:09:47.903 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-13T02:09:47.903 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-13T02:09:47.903 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-13T02:09:47.903 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-13T02:09:47.903 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.row_cache: None 2023-07-13T02:09:47.904 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.wal_filter: None 2023-07-13T02:09:47.904 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-13T02:09:47.904 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.allow_ingest_behind: 0 2023-07-13T02:09:47.904 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.two_write_queues: 0 2023-07-13T02:09:47.904 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.manual_wal_flush: 0 2023-07-13T02:09:47.904 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.wal_compression: 0 2023-07-13T02:09:47.904 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.atomic_flush: 0 2023-07-13T02:09:47.905 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-13T02:09:47.905 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-13T02:09:47.905 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-13T02:09:47.905 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.log_readahead_size: 0 2023-07-13T02:09:47.905 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-13T02:09:47.905 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.best_efforts_recovery: 0 2023-07-13T02:09:47.905 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-13T02:09:47.906 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-13T02:09:47.906 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.allow_data_in_errors: 0 2023-07-13T02:09:47.906 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.db_host_id: __hostname__ 2023-07-13T02:09:47.906 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-13T02:09:47.906 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_background_jobs: 2 2023-07-13T02:09:47.906 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_background_compactions: -1 2023-07-13T02:09:47.906 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_subcompactions: 1 2023-07-13T02:09:47.906 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-13T02:09:47.907 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-13T02:09:47.907 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-13T02:09:47.907 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_total_wal_size: 0 2023-07-13T02:09:47.907 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-13T02:09:47.907 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-13T02:09:47.907 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-13T02:09:47.907 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-13T02:09:47.908 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_open_files: -1 2023-07-13T02:09:47.908 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.bytes_per_sync: 0 2023-07-13T02:09:47.908 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-13T02:09:47.908 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-13T02:09:47.908 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.compaction_readahead_size: 0 2023-07-13T02:09:47.908 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_background_flushes: -1 2023-07-13T02:09:47.908 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Compression algorithms supported: 2023-07-13T02:09:47.909 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-13T02:09:47.909 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: kZSTD supported: 0 2023-07-13T02:09:47.909 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: kXpressCompression supported: 0 2023-07-13T02:09:47.909 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: kLZ4HCCompression supported: 1 2023-07-13T02:09:47.909 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: kZlibCompression supported: 1 2023-07-13T02:09:47.909 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: kSnappyCompression supported: 1 2023-07-13T02:09:47.909 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: kLZ4Compression supported: 1 2023-07-13T02:09:47.909 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: kBZip2Compression supported: 0 2023-07-13T02:09:47.910 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-13T02:09:47.910 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-13T02:09:47.910 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-07-13T02:09:47.910 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-13T02:09:47.910 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-13T02:09:47.910 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.merge_operator: 2023-07-13T02:09:47.910 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.compaction_filter: None 2023-07-13T02:09:47.910 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.compaction_filter_factory: None 2023-07-13T02:09:47.911 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.sst_partitioner_factory: None 2023-07-13T02:09:47.911 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-13T02:09:47.911 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-13T02:09:47.911 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a52f5ef4e0) 2023-07-13T02:09:47.911 INFO:journalctl@ceph.mon.c.smithi032.stdout: cache_index_and_filter_blocks: 1 2023-07-13T02:09:47.911 INFO:journalctl@ceph.mon.c.smithi032.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-13T02:09:47.911 INFO:journalctl@ceph.mon.c.smithi032.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-13T02:09:47.912 INFO:journalctl@ceph.mon.c.smithi032.stdout: pin_top_level_index_and_filter: 1 2023-07-13T02:09:47.912 INFO:journalctl@ceph.mon.c.smithi032.stdout: index_type: 0 2023-07-13T02:09:47.912 INFO:journalctl@ceph.mon.c.smithi032.stdout: data_block_index_type: 0 2023-07-13T02:09:47.912 INFO:journalctl@ceph.mon.c.smithi032.stdout: index_shortening: 1 2023-07-13T02:09:47.912 INFO:journalctl@ceph.mon.c.smithi032.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-13T02:09:47.912 INFO:journalctl@ceph.mon.c.smithi032.stdout: checksum: 4 2023-07-13T02:09:47.912 INFO:journalctl@ceph.mon.c.smithi032.stdout: no_block_cache: 0 2023-07-13T02:09:47.913 INFO:journalctl@ceph.mon.c.smithi032.stdout: block_cache: 0x55a52e901350 2023-07-13T02:09:47.913 INFO:journalctl@ceph.mon.c.smithi032.stdout: block_cache_name: BinnedLRUCache 2023-07-13T02:09:47.913 INFO:journalctl@ceph.mon.c.smithi032.stdout: block_cache_options: 2023-07-13T02:09:47.913 INFO:journalctl@ceph.mon.c.smithi032.stdout: capacity : 536870912 2023-07-13T02:09:47.913 INFO:journalctl@ceph.mon.c.smithi032.stdout: num_shard_bits : 4 2023-07-13T02:09:47.913 INFO:journalctl@ceph.mon.c.smithi032.stdout: strict_capacity_limit : 0 2023-07-13T02:09:47.913 INFO:journalctl@ceph.mon.c.smithi032.stdout: high_pri_pool_ratio: 0.000 2023-07-13T02:09:47.913 INFO:journalctl@ceph.mon.c.smithi032.stdout: block_cache_compressed: (nil) 2023-07-13T02:09:47.914 INFO:journalctl@ceph.mon.c.smithi032.stdout: persistent_cache: (nil) 2023-07-13T02:09:47.914 INFO:journalctl@ceph.mon.c.smithi032.stdout: block_size: 4096 2023-07-13T02:09:47.914 INFO:journalctl@ceph.mon.c.smithi032.stdout: block_size_deviation: 10 2023-07-13T02:09:47.914 INFO:journalctl@ceph.mon.c.smithi032.stdout: block_restart_interval: 16 2023-07-13T02:09:47.914 INFO:journalctl@ceph.mon.c.smithi032.stdout: index_block_restart_interval: 1 2023-07-13T02:09:47.914 INFO:journalctl@ceph.mon.c.smithi032.stdout: metadata_block_size: 4096 2023-07-13T02:09:47.914 INFO:journalctl@ceph.mon.c.smithi032.stdout: partition_filters: 0 2023-07-13T02:09:47.914 INFO:journalctl@ceph.mon.c.smithi032.stdout: use_delta_encoding: 1 2023-07-13T02:09:47.915 INFO:journalctl@ceph.mon.c.smithi032.stdout: filter_policy: bloomfilter 2023-07-13T02:09:47.915 INFO:journalctl@ceph.mon.c.smithi032.stdout: whole_key_filtering: 1 2023-07-13T02:09:47.915 INFO:journalctl@ceph.mon.c.smithi032.stdout: verify_compression: 0 2023-07-13T02:09:47.915 INFO:journalctl@ceph.mon.c.smithi032.stdout: read_amp_bytes_per_bit: 0 2023-07-13T02:09:47.915 INFO:journalctl@ceph.mon.c.smithi032.stdout: format_version: 5 2023-07-13T02:09:47.915 INFO:journalctl@ceph.mon.c.smithi032.stdout: enable_index_compression: 1 2023-07-13T02:09:47.915 INFO:journalctl@ceph.mon.c.smithi032.stdout: block_align: 0 2023-07-13T02:09:47.915 INFO:journalctl@ceph.mon.c.smithi032.stdout: max_auto_readahead_size: 262144 2023-07-13T02:09:47.916 INFO:journalctl@ceph.mon.c.smithi032.stdout: prepopulate_block_cache: 0 2023-07-13T02:09:47.916 INFO:journalctl@ceph.mon.c.smithi032.stdout: initial_auto_readahead_size: 8192 2023-07-13T02:09:47.916 INFO:journalctl@ceph.mon.c.smithi032.stdout: num_file_reads_for_auto_readahead: 2 2023-07-13T02:09:47.916 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.write_buffer_size: 33554432 2023-07-13T02:09:47.916 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_write_buffer_number: 2 2023-07-13T02:09:47.916 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.compression: NoCompression 2023-07-13T02:09:47.916 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.bottommost_compression: Disabled 2023-07-13T02:09:47.917 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.prefix_extractor: nullptr 2023-07-13T02:09:47.917 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-13T02:09:47.917 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.num_levels: 7 2023-07-13T02:09:47.917 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-13T02:09:47.917 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-13T02:09:47.917 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-13T02:09:47.917 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-13T02:09:47.918 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-13T02:09:47.918 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-13T02:09:47.918 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-13T02:09:47.918 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-13T02:09:47.919 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-13T02:09:47.919 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-13T02:09:47.919 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-13T02:09:47.919 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-13T02:09:47.919 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-13T02:09:47.919 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.compression_opts.level: 32767 2023-07-13T02:09:47.919 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.compression_opts.strategy: 0 2023-07-13T02:09:47.920 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-13T02:09:47.920 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-13T02:09:47.920 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-13T02:09:47.920 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-13T02:09:47.920 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.compression_opts.enabled: false 2023-07-13T02:09:47.920 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-13T02:09:47.920 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-13T02:09:47.921 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-13T02:09:47.921 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-13T02:09:47.921 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.target_file_size_base: 67108864 2023-07-13T02:09:47.921 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-13T02:09:47.921 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-13T02:09:47.921 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-13T02:09:47.921 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-13T02:09:47.921 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-13T02:09:47.922 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-13T02:09:47.922 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-13T02:09:47.922 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-13T02:09:47.922 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-13T02:09:47.922 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-13T02:09:47.922 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-13T02:09:47.922 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-13T02:09:47.923 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-13T02:09:47.923 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-13T02:09:47.923 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.arena_block_size: 1048576 2023-07-13T02:09:47.923 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-13T02:09:47.923 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-13T02:09:47.923 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.disable_auto_compactions: 0 2023-07-13T02:09:47.923 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-13T02:09:47.923 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-13T02:09:47.924 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-13T02:09:47.924 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-13T02:09:47.924 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-13T02:09:47.924 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-13T02:09:47.924 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-13T02:09:47.924 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-13T02:09:47.924 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-13T02:09:47.925 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-13T02:09:47.925 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.table_properties_collectors: 2023-07-13T02:09:47.925 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.inplace_update_support: 0 2023-07-13T02:09:47.925 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-13T02:09:47.925 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-13T02:09:47.925 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-13T02:09:47.925 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-13T02:09:47.926 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.bloom_locality: 0 2023-07-13T02:09:47.926 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.max_successive_merges: 0 2023-07-13T02:09:47.926 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-13T02:09:47.926 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.paranoid_file_checks: 0 2023-07-13T02:09:47.926 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.force_consistency_checks: 1 2023-07-13T02:09:47.926 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.report_bg_io_stats: 0 2023-07-13T02:09:47.926 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.ttl: 2592000 2023-07-13T02:09:47.926 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-13T02:09:47.927 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-13T02:09:47.927 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-13T02:09:47.927 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.enable_blob_files: false 2023-07-13T02:09:47.927 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.min_blob_size: 0 2023-07-13T02:09:47.927 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.blob_file_size: 268435456 2023-07-13T02:09:47.927 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-13T02:09:47.927 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-13T02:09:47.928 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-13T02:09:47.928 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-13T02:09:47.928 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-13T02:09:47.928 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.blob_file_starting_level: 0 2023-07-13T02:09:47.928 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-13T02:09:47.928 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-13T02:09:47.928 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-13T02:09:47.928 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9a8e909b-ea1d-48b4-a8ac-75a208b3531d 2023-07-13T02:09:47.929 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689214187452282, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-13T02:09:47.929 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-13T02:09:47.929 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689214187452871, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689214187, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9a8e909b-ea1d-48b4-a8ac-75a208b3531d", "db_session_id": "Q0OA1WF7JR16NS9Q0B7T", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-13T02:09:47.929 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689214187452969, "job": 1, "event": "recovery_finished"} 2023-07-13T02:09:47.929 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-13T02:09:47.929 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-13T02:09:47.929 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a52e99e000 2023-07-13T02:09:47.930 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: DB pointer 0x55a52e98a000 2023-07-13T02:09:47.930 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-13T02:09:47.930 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-13T02:09:47.930 INFO:journalctl@ceph.mon.c.smithi032.stdout: ** DB Stats ** 2023-07-13T02:09:47.930 INFO:journalctl@ceph.mon.c.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-13T02:09:47.930 INFO:journalctl@ceph.mon.c.smithi032.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-13T02:09:47.930 INFO:journalctl@ceph.mon.c.smithi032.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-13T02:09:47.931 INFO:journalctl@ceph.mon.c.smithi032.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-13T02:09:47.931 INFO:journalctl@ceph.mon.c.smithi032.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-13T02:09:47.931 INFO:journalctl@ceph.mon.c.smithi032.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-13T02:09:47.931 INFO:journalctl@ceph.mon.c.smithi032.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-13T02:09:47.931 INFO:journalctl@ceph.mon.c.smithi032.stdout: 2023-07-13T02:09:47.932 INFO:journalctl@ceph.mon.c.smithi032.stdout: ** Compaction Stats [default] ** 2023-07-13T02:09:47.932 INFO:journalctl@ceph.mon.c.smithi032.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-13T02:09:47.932 INFO:journalctl@ceph.mon.c.smithi032.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-13T02:09:47.932 INFO:journalctl@ceph.mon.c.smithi032.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-13T02:09:47.932 INFO:journalctl@ceph.mon.c.smithi032.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-13T02:09:47.932 INFO:journalctl@ceph.mon.c.smithi032.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-13T02:09:47.932 INFO:journalctl@ceph.mon.c.smithi032.stdout: 2023-07-13T02:09:47.933 INFO:journalctl@ceph.mon.c.smithi032.stdout: ** Compaction Stats [default] ** 2023-07-13T02:09:47.933 INFO:journalctl@ceph.mon.c.smithi032.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-13T02:09:47.933 INFO:journalctl@ceph.mon.c.smithi032.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-13T02:09:47.933 INFO:journalctl@ceph.mon.c.smithi032.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-13T02:09:47.933 INFO:journalctl@ceph.mon.c.smithi032.stdout: 2023-07-13T02:09:47.933 INFO:journalctl@ceph.mon.c.smithi032.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-13T02:09:47.933 INFO:journalctl@ceph.mon.c.smithi032.stdout: 2023-07-13T02:09:47.934 INFO:journalctl@ceph.mon.c.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-13T02:09:47.934 INFO:journalctl@ceph.mon.c.smithi032.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-13T02:09:47.934 INFO:journalctl@ceph.mon.c.smithi032.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-13T02:09:47.934 INFO:journalctl@ceph.mon.c.smithi032.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-13T02:09:47.934 INFO:journalctl@ceph.mon.c.smithi032.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-13T02:09:47.934 INFO:journalctl@ceph.mon.c.smithi032.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-13T02:09:47.934 INFO:journalctl@ceph.mon.c.smithi032.stdout: Cumulative compaction: 0.00 GB write, 0.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-13T02:09:47.935 INFO:journalctl@ceph.mon.c.smithi032.stdout: Interval compaction: 0.00 GB write, 0.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-13T02:09:47.935 INFO:journalctl@ceph.mon.c.smithi032.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-13T02:09:47.935 INFO:journalctl@ceph.mon.c.smithi032.stdout: Block cache BinnedLRUCache@0x55a52e901350#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2023-07-13T02:09:47.935 INFO:journalctl@ceph.mon.c.smithi032.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-13T02:09:47.935 INFO:journalctl@ceph.mon.c.smithi032.stdout: 2023-07-13T02:09:47.935 INFO:journalctl@ceph.mon.c.smithi032.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-13T02:09:47.935 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-07-13T02:09:47.935 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: using public_addrv [v2:172.21.15.32:3301/0,v1:172.21.15.32:6790/0] 2023-07-13T02:09:47.936 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: starting mon.c rank -1 at public addrs [v2:172.21.15.32:3301/0,v1:172.21.15.32:6790/0] at bind addrs [v2:172.21.15.32:3301/0,v1:172.21.15.32:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 11418e3c-2122-11ee-9b31-001a4aab830c 2023-07-13T02:09:47.936 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mon.c@-1(???) e0 preinit fsid 11418e3c-2122-11ee-9b31-001a4aab830c 2023-07-13T02:09:47.936 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mon.c@-1(synchronizing).mds e1 new map 2023-07-13T02:09:47.936 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mon.c@-1(synchronizing).mds e1 print_map 2023-07-13T02:09:47.936 INFO:journalctl@ceph.mon.c.smithi032.stdout: e1 2023-07-13T02:09:47.936 INFO:journalctl@ceph.mon.c.smithi032.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-13T02:09:47.936 INFO:journalctl@ceph.mon.c.smithi032.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-13T02:09:47.937 INFO:journalctl@ceph.mon.c.smithi032.stdout: legacy client fscid: -1 2023-07-13T02:09:47.937 INFO:journalctl@ceph.mon.c.smithi032.stdout: 2023-07-13T02:09:47.937 INFO:journalctl@ceph.mon.c.smithi032.stdout: No filesystems configured 2023-07-13T02:09:47.937 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-13T02:09:47.937 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-13T02:09:47.937 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-13T02:09:47.937 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-13T02:09:47.937 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-13T02:09:47.938 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-13T02:09:47.938 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-13T02:09:47.938 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-13T02:09:47.938 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-13T02:09:47.938 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-13T02:09:47.938 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mkfs 11418e3c-2122-11ee-9b31-001a4aab830c 2023-07-13T02:09:47.938 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-13T02:09:47.939 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-13T02:09:47.939 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: monmap e1: 1 mons at {a=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0]} removed_ranks: {} 2023-07-13T02:09:47.939 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: fsmap 2023-07-13T02:09:47.939 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: osdmap e1: 0 total, 0 up, 0 in 2023-07-13T02:09:47.939 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mgrmap e1: no daemons active 2023-07-13T02:09:47.939 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3934702177' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-07-13T02:09:47.939 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1845941787' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-13T02:09:47.939 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1845941787' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-07-13T02:09:47.940 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3292780349' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:47.940 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-13T02:09:47.940 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: monmap e1: 1 mons at {a=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0]} removed_ranks: {} 2023-07-13T02:09:47.940 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: fsmap 2023-07-13T02:09:47.940 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: osdmap e1: 0 total, 0 up, 0 in 2023-07-13T02:09:47.940 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mgrmap e1: no daemons active 2023-07-13T02:09:47.940 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1467726919' entity='client.admin' 2023-07-13T02:09:47.941 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/4067414552' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-13T02:09:47.941 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3832074614' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-13T02:09:47.941 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3464646265' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-13T02:09:47.941 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Activating manager daemon y 2023-07-13T02:09:47.941 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mgrmap e2: y(active, starting, since 0.00293762s) 2023-07-13T02:09:47.941 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14100 172.21.15.32:0/1862309846' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-13T02:09:47.941 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14100 172.21.15.32:0/1862309846' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-13T02:09:47.942 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14100 172.21.15.32:0/1862309846' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-13T02:09:47.942 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14100 172.21.15.32:0/1862309846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-13T02:09:47.942 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14100 172.21.15.32:0/1862309846' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-13T02:09:47.942 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Manager daemon y is now available 2023-07-13T02:09:47.942 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14100 172.21.15.32:0/1862309846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-13T02:09:47.942 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14100 172.21.15.32:0/1862309846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-13T02:09:47.942 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14100 172.21.15.32:0/1862309846' entity='mgr.y' 2023-07-13T02:09:47.942 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14100 172.21.15.32:0/1862309846' entity='mgr.y' 2023-07-13T02:09:47.943 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14100 172.21.15.32:0/1862309846' entity='mgr.y' 2023-07-13T02:09:47.943 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mgrmap e3: y(active, since 1.00705s) 2023-07-13T02:09:47.943 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3553713678' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-13T02:09:47.943 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mgrmap e4: y(active, since 3s) 2023-07-13T02:09:47.943 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1237794827' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-13T02:09:47.943 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1237794827' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-07-13T02:09:47.943 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2520975079' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-07-13T02:09:47.943 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2520975079' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-07-13T02:09:47.944 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mgrmap e5: y(active, since 5s) 2023-07-13T02:09:47.944 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2441822803' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-13T02:09:47.944 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Active manager daemon y restarted 2023-07-13T02:09:47.944 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Activating manager daemon y 2023-07-13T02:09:47.944 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: osdmap e2: 0 total, 0 up, 0 in 2023-07-13T02:09:47.944 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mgrmap e6: y(active, starting, since 0.00325692s) 2023-07-13T02:09:47.944 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-13T02:09:47.945 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-13T02:09:47.945 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-13T02:09:47.945 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-13T02:09:47.945 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-13T02:09:47.945 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Manager daemon y is now available 2023-07-13T02:09:47.945 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Found migration_current of "None". Setting to last migration. 2023-07-13T02:09:47.945 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:09:47.946 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:09:47.946 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:47.946 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:47.946 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:47.946 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-13T02:09:47.946 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-13T02:09:47.946 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:09:47.946 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:09:47.947 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-13T02:09:47.947 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mgrmap e7: y(active, since 1.00535s) 2023-07-13T02:09:47.947 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-13T02:09:47.947 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:47.947 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: [13/Jul/2023:02:08:56] ENGINE Bus STARTING 2023-07-13T02:09:47.947 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: [13/Jul/2023:02:08:56] ENGINE Serving on http://172.21.15.32:8765 2023-07-13T02:09:47.947 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: [13/Jul/2023:02:08:56] ENGINE Serving on https://172.21.15.32:7150 2023-07-13T02:09:47.948 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: [13/Jul/2023:02:08:56] ENGINE Bus STARTED 2023-07-13T02:09:47.948 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mgrmap e8: y(active, since 3s) 2023-07-13T02:09:47.948 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:09:47.948 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:47.948 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:47.948 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:47.948 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:09:47.949 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:09:47.949 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:47.949 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Generating ssh key... 2023-07-13T02:09:47.949 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:47.949 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi032", "addr": "172.21.15.32", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:47.949 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Deploying cephadm binary to smithi032 2023-07-13T02:09:47.949 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:09:47.950 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Added host smithi032 2023-07-13T02:09:47.950 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:47.950 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:09:47.950 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:47.950 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Saving service mon spec with placement count:5 2023-07-13T02:09:47.950 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:09:47.950 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:09:47.951 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:47.951 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Saving service mgr spec with placement count:2 2023-07-13T02:09:47.951 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:09:47.951 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2150654459' entity='client.admin' 2023-07-13T02:09:47.951 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:09:47.951 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3403984800' entity='client.admin' 2023-07-13T02:09:47.951 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14120 172.21.15.32:0/1498958878' entity='mgr.y' 2023-07-13T02:09:47.952 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/13468188' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-07-13T02:09:47.952 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/13468188' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-07-13T02:09:47.952 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mgrmap e9: y(active, since 14s) 2023-07-13T02:09:47.952 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/982001257' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-13T02:09:47.952 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Active manager daemon y restarted 2023-07-13T02:09:47.952 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Activating manager daemon y 2023-07-13T02:09:47.952 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: osdmap e3: 0 total, 0 up, 0 in 2023-07-13T02:09:47.952 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mgrmap e10: y(active, starting, since 0.00347432s) 2023-07-13T02:09:47.953 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-13T02:09:47.953 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-13T02:09:47.953 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-13T02:09:47.953 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-13T02:09:47.953 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-13T02:09:47.953 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Manager daemon y is now available 2023-07-13T02:09:47.953 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:47.954 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:47.954 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-07-13T02:09:47.954 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-07-13T02:09:47.954 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.954 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mgrmap e11: y(active, since 1.0056s) 2023-07-13T02:09:47.954 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: [13/Jul/2023:02:09:18] ENGINE Bus STARTING 2023-07-13T02:09:47.954 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.954 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.955 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.955 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: [13/Jul/2023:02:09:18] ENGINE Serving on http://172.21.15.32:8765 2023-07-13T02:09:47.955 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: [13/Jul/2023:02:09:18] ENGINE Serving on https://172.21.15.32:7150 2023-07-13T02:09:47.955 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: [13/Jul/2023:02:09:18] ENGINE Bus STARTED 2023-07-13T02:09:47.955 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:47.955 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mgrmap e12: y(active, since 2s) 2023-07-13T02:09:47.956 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:47.956 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.956 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/654036068' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-07-13T02:09:47.956 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mgrmap e13: y(active, since 6s) 2023-07-13T02:09:47.956 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1608105501' entity='client.admin' 2023-07-13T02:09:47.956 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/890431937' entity='client.admin' 2023-07-13T02:09:47.956 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.957 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.957 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi032", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:09:47.957 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.957 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:47.957 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.957 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:47.957 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.957 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:47.958 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:47.958 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:09:47.958 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Updating smithi032:/etc/ceph/ceph.conf 2023-07-13T02:09:47.958 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Updating smithi032:/var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-13T02:09:47.958 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2023-07-13T02:09:47.958 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Updating smithi032:/var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/config/ceph.client.admin.keyring 2023-07-13T02:09:47.958 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.959 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.959 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.959 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi191", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:47.959 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Deploying cephadm binary to smithi191 2023-07-13T02:09:47.959 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.959 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:47.959 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Added host smithi191 2023-07-13T02:09:47.959 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.960 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.960 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.960 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-13T02:09:47.960 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3984784913' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-13T02:09:47.960 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.960 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.960 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.961 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.961 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:09:47.961 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:47.961 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:09:47.961 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Updating smithi191:/etc/ceph/ceph.conf 2023-07-13T02:09:47.961 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Updating smithi191:/var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-13T02:09:47.961 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2023-07-13T02:09:47.962 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3984784913' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-13T02:09:47.962 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: osdmap e4: 0 total, 0 up, 0 in 2023-07-13T02:09:47.962 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Updating smithi191:/var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/config/ceph.client.admin.keyring 2023-07-13T02:09:47.962 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.962 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.962 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.962 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:47.962 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.963 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:47.963 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:47.963 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:09:47.963 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.963 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-13T02:09:47.963 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:47.963 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi032:172.21.15.32=a;smithi032:[v2:172.21.15.32:3301,v1:172.21.15.32:6790]=c;smithi191:172.21.15.191=b", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:47.963 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Saving service mon spec with placement smithi032:172.21.15.32=a;smithi032:[v2:172.21.15.32:3301,v1:172.21.15.32:6790]=c;smithi191:172.21.15.191=b;count:3 2023-07-13T02:09:47.964 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Deploying daemon mon.b on smithi191 2023-07-13T02:09:47.964 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:47.964 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-13T02:09:47.964 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mon.a calling monitor election 2023-07-13T02:09:47.964 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:47.964 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:47.964 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:47.965 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:47.965 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mon.b calling monitor election 2023-07-13T02:09:47.965 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:47.965 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:47.965 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:47.965 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:47.965 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:47.965 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2023-07-13T02:09:47.966 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: monmap e2: 2 mons at {a=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0],b=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0]} removed_ranks: {} 2023-07-13T02:09:47.966 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: fsmap 2023-07-13T02:09:47.966 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: osdmap e4: 0 total, 0 up, 0 in 2023-07-13T02:09:47.966 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mgrmap e13: y(active, since 28s) 2023-07-13T02:09:47.966 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: overall HEALTH_OK 2023-07-13T02:09:47.966 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.966 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.967 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:47.967 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-13T02:09:47.967 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:47.967 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: Deploying daemon mon.c on smithi032 2023-07-13T02:09:47.967 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.191:0/930590995' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-13T02:09:47.967 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:47.967 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-13T02:09:47.968 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: mon.c@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-07-13T02:09:47.968 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: expand_channel_meta expand map: {default=false} 2023-07-13T02:09:47.968 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: expand_channel_meta from 'false' to 'false' 2023-07-13T02:09:47.968 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: expand_channel_meta expanded map: {default=false} 2023-07-13T02:09:47.968 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: expand_channel_meta expand map: {default=info} 2023-07-13T02:09:47.968 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: expand_channel_meta from 'info' to 'info' 2023-07-13T02:09:47.968 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: expand_channel_meta expanded map: {default=info} 2023-07-13T02:09:47.969 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: expand_channel_meta expand map: {default=daemon} 2023-07-13T02:09:47.969 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-13T02:09:47.969 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: expand_channel_meta expanded map: {default=daemon} 2023-07-13T02:09:47.969 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: expand_channel_meta expand map: {default=debug} 2023-07-13T02:09:47.969 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: expand_channel_meta from 'debug' to 'debug' 2023-07-13T02:09:47.969 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:47 smithi032 ceph-mon[116810]: expand_channel_meta expanded map: {default=debug} 2023-07-13T02:09:52.887 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[104859]: mon.a calling monitor election 2023-07-13T02:09:52.887 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[104859]: mon.b calling monitor election 2023-07-13T02:09:52.887 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-13T02:09:52.887 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:52.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-13T02:09:52.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-13T02:09:52.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[104859]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:52.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-13T02:09:52.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[104859]: mon.c calling monitor election 2023-07-13T02:09:52.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-13T02:09:52.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[104859]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:52.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-13T02:09:52.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-13T02:09:52.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[104859]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2023-07-13T02:09:52.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[104859]: monmap e3: 3 mons at {a=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0],b=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0],c=[v2:172.21.15.32:3301/0,v1:172.21.15.32:6790/0]} removed_ranks: {} 2023-07-13T02:09:52.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[104859]: fsmap 2023-07-13T02:09:52.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[104859]: osdmap e4: 0 total, 0 up, 0 in 2023-07-13T02:09:52.890 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[104859]: mgrmap e13: y(active, since 35s) 2023-07-13T02:09:52.890 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[104859]: overall HEALTH_OK 2023-07-13T02:09:52.890 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:52.890 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:52.890 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[116810]: mon.a calling monitor election 2023-07-13T02:09:52.890 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[116810]: mon.b calling monitor election 2023-07-13T02:09:52.891 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-13T02:09:52.891 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:52.891 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-13T02:09:52.891 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-13T02:09:52.891 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[116810]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:52.891 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-13T02:09:52.891 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[116810]: mon.c calling monitor election 2023-07-13T02:09:52.892 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-13T02:09:52.892 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[116810]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:52.892 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-13T02:09:52.892 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-13T02:09:52.892 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[116810]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2023-07-13T02:09:52.892 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[116810]: monmap e3: 3 mons at {a=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0],b=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0],c=[v2:172.21.15.32:3301/0,v1:172.21.15.32:6790/0]} removed_ranks: {} 2023-07-13T02:09:52.893 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[116810]: fsmap 2023-07-13T02:09:52.893 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[116810]: osdmap e4: 0 total, 0 up, 0 in 2023-07-13T02:09:52.893 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[116810]: mgrmap e13: y(active, since 35s) 2023-07-13T02:09:52.893 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[116810]: overall HEALTH_OK 2023-07-13T02:09:52.893 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:52.894 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:52 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:52.980 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:52 smithi191 ceph-mon[108588]: mon.a calling monitor election 2023-07-13T02:09:52.980 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:52 smithi191 ceph-mon[108588]: mon.b calling monitor election 2023-07-13T02:09:52.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:52 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-13T02:09:52.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:52 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-13T02:09:52.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:52 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-13T02:09:52.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:52 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-13T02:09:52.982 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:52 smithi191 ceph-mon[108588]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:52.982 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:52 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-13T02:09:52.982 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:52 smithi191 ceph-mon[108588]: mon.c calling monitor election 2023-07-13T02:09:52.982 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:52 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-13T02:09:52.982 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:52 smithi191 ceph-mon[108588]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:52.982 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:52 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-13T02:09:52.982 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:52 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-13T02:09:52.983 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:52 smithi191 ceph-mon[108588]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2023-07-13T02:09:52.983 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:52 smithi191 ceph-mon[108588]: monmap e3: 3 mons at {a=[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0],b=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0],c=[v2:172.21.15.32:3301/0,v1:172.21.15.32:6790/0]} removed_ranks: {} 2023-07-13T02:09:52.983 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:52 smithi191 ceph-mon[108588]: fsmap 2023-07-13T02:09:52.983 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:52 smithi191 ceph-mon[108588]: osdmap e4: 0 total, 0 up, 0 in 2023-07-13T02:09:52.983 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:52 smithi191 ceph-mon[108588]: mgrmap e13: y(active, since 35s) 2023-07-13T02:09:52.983 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:52 smithi191 ceph-mon[108588]: overall HEALTH_OK 2023-07-13T02:09:52.983 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:52 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:52.984 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:52 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:53.231 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-13T02:09:53.232 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":3,"fsid":"11418e3c-2122-11ee-9b31-001a4aab830c","modified":"2023-07-13T02:09:47.478776Z","created":"2023-07-13T02:08:30.011732Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3301","nonce":0},{"type":"v1","addr":"172.21.15.32:6790","nonce":0}]},"addr":"172.21.15.32:6790/0","public_addr":"172.21.15.32:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2023-07-13T02:09:53.234 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 3 2023-07-13T02:09:53.653 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-07-13T02:09:53.654 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph config generate-minimal-conf 2023-07-13T02:09:53.886 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:53 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:53.887 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:53 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:53.887 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:53 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:53.887 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:53 smithi032 ceph-mon[116810]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:53.887 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:53 smithi032 ceph-mon[116810]: from='client.? 172.21.15.191:0/619244745' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-13T02:09:53.887 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:53 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:53.887 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:53 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:53.887 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:53 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-13T02:09:53.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:53 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:53.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:53 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:53.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:53 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:53.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:53 smithi032 ceph-mon[104859]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:53.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:53 smithi032 ceph-mon[104859]: from='client.? 172.21.15.191:0/619244745' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-13T02:09:53.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:53 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:53.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:53 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:53.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:53 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-13T02:09:53.894 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:09:53.980 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:53 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:53.980 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:53 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:53.980 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:53 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:53.980 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:53 smithi191 ceph-mon[108588]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:53.980 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:53 smithi191 ceph-mon[108588]: from='client.? 172.21.15.191:0/619244745' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-13T02:09:53.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:53 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:53.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:53 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:53.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:53 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-13T02:09:54.594 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:54.594 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:54.594 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:54.595 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:09:54.595 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[116810]: Updating smithi032:/etc/ceph/ceph.conf 2023-07-13T02:09:54.595 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[116810]: Updating smithi191:/etc/ceph/ceph.conf 2023-07-13T02:09:54.595 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[116810]: Updating smithi032:/var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-13T02:09:54.595 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[116810]: Updating smithi191:/var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-13T02:09:54.595 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:54.595 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:54.596 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:54.596 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:54.596 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:54.596 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:54.596 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:54.596 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:54.597 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:09:54.597 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[104859]: Updating smithi032:/etc/ceph/ceph.conf 2023-07-13T02:09:54.597 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[104859]: Updating smithi191:/etc/ceph/ceph.conf 2023-07-13T02:09:54.597 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[104859]: Updating smithi032:/var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-13T02:09:54.597 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[104859]: Updating smithi191:/var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-13T02:09:54.597 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:54.597 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:54.597 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:54.598 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:54.598 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:54.598 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-13T02:09:54.598 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-13T02:09:54.598 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:54.886 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-13T02:09:54.887 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-13T02:09:54.887 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:54 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:54.980 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:54 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:54.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:54 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:54.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:54 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:54.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:54 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:09:54.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:54 smithi191 ceph-mon[108588]: Updating smithi032:/etc/ceph/ceph.conf 2023-07-13T02:09:54.982 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:54 smithi191 ceph-mon[108588]: Updating smithi191:/etc/ceph/ceph.conf 2023-07-13T02:09:54.982 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:54 smithi191 ceph-mon[108588]: Updating smithi032:/var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-13T02:09:54.982 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:54 smithi191 ceph-mon[108588]: Updating smithi191:/var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/config/ceph.conf 2023-07-13T02:09:54.983 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:54 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:54.983 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:54 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:54.983 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:54 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:54.983 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:54 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:54.984 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:54 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:54.984 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:54 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-13T02:09:54.984 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:54 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-13T02:09:54.985 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:54 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:55.048 INFO:teuthology.orchestra.run.smithi032.stdout:# minimal ceph.conf for 11418e3c-2122-11ee-9b31-001a4aab830c 2023-07-13T02:09:55.048 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2023-07-13T02:09:55.048 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = 11418e3c-2122-11ee-9b31-001a4aab830c 2023-07-13T02:09:55.048 INFO:teuthology.orchestra.run.smithi032.stdout: mon_host = [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] [v2:172.21.15.32:3301/0,v1:172.21.15.32:6790/0] 2023-07-13T02:09:55.497 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-07-13T02:09:55.498 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2023-07-13T02:09:55.498 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.conf 2023-07-13T02:09:55.529 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2023-07-13T02:09:55.529 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-13T02:09:55.600 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-07-13T02:09:55.600 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.conf 2023-07-13T02:09:55.638 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-07-13T02:09:55.638 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-13T02:09:55.710 INFO:tasks.cephadm:Adding mgr.y on smithi032 2023-07-13T02:09:55.710 INFO:tasks.cephadm:Adding mgr.x on smithi191 2023-07-13T02:09:55.711 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph orch apply mgr '2;smithi032=y;smithi191=x' 2023-07-13T02:09:55.887 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:55 smithi032 ceph-mon[104859]: Reconfiguring mon.a (unknown last config time)... 2023-07-13T02:09:55.887 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:55 smithi032 ceph-mon[104859]: Reconfiguring daemon mon.a on smithi032 2023-07-13T02:09:55.887 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:55 smithi032 ceph-mon[104859]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:55.887 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:55 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/476981921' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:55.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:55 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:55.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:55 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:55.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:55 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-13T02:09:55.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:55 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-13T02:09:55.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:55 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:55.888 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:55 smithi032 ceph-mon[116810]: Reconfiguring mon.a (unknown last config time)... 2023-07-13T02:09:55.889 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:55 smithi032 ceph-mon[116810]: Reconfiguring daemon mon.a on smithi032 2023-07-13T02:09:55.889 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:55 smithi032 ceph-mon[116810]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:55.889 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:55 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/476981921' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:55.889 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:55 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:55.889 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:55 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:55.889 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:55 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-13T02:09:55.889 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:55 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-13T02:09:55.890 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:55 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:55.971 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.b/config 2023-07-13T02:09:55.980 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:55 smithi191 ceph-mon[108588]: Reconfiguring mon.a (unknown last config time)... 2023-07-13T02:09:55.980 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:55 smithi191 ceph-mon[108588]: Reconfiguring daemon mon.a on smithi032 2023-07-13T02:09:55.980 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:55 smithi191 ceph-mon[108588]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:55.980 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:55 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/476981921' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:55.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:55 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:55.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:55 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:55.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:55 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-13T02:09:55.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:55 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-13T02:09:55.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:55 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:57.230 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:57 smithi191 ceph-mon[108588]: Reconfiguring mon.c (monmap changed)... 2023-07-13T02:09:57.230 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:57 smithi191 ceph-mon[108588]: Reconfiguring daemon mon.c on smithi032 2023-07-13T02:09:57.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:57 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:57.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:57 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:57.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:57 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-13T02:09:57.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:57 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-13T02:09:57.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:57 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:57.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:57 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:57.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:57 smithi032 ceph-mon[104859]: Reconfiguring mon.c (monmap changed)... 2023-07-13T02:09:57.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:57 smithi032 ceph-mon[104859]: Reconfiguring daemon mon.c on smithi032 2023-07-13T02:09:57.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:57 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:57.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:57 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:57.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:57 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-13T02:09:57.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:57 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-13T02:09:57.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:57 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:57.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:57 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:57.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:57 smithi032 ceph-mon[116810]: Reconfiguring mon.c (monmap changed)... 2023-07-13T02:09:57.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:57 smithi032 ceph-mon[116810]: Reconfiguring daemon mon.c on smithi032 2023-07-13T02:09:57.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:57 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:57.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:57 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:57.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:57 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-13T02:09:57.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:57 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-13T02:09:57.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:57 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:57.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:57 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:57.416 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mgr update... 2023-07-13T02:09:57.809 DEBUG:teuthology.orchestra.run.smithi191:mgr.x> sudo journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mgr.x.service 2023-07-13T02:09:57.811 INFO:tasks.cephadm:Deploying OSDs... 2023-07-13T02:09:57.811 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2023-07-13T02:09:57.811 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/scratch_devs of=/dev/stdout 2023-07-13T02:09:57.834 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2023-07-13T02:09:57.834 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/nvme1n1 2023-07-13T02:09:57.842 INFO:journalctl@ceph.mgr.x.smithi191.stdout:-- Logs begin at Thu 2023-07-13 01:53:22 UTC. -- 2023-07-13T02:09:57.897 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/nvme1n1 2023-07-13T02:09:57.897 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-07-13T02:09:57.897 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 421164 Links: 1 Device type: 103,1 2023-07-13T02:09:57.898 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-07-13T02:09:57.898 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:nvme_device_t:s0 2023-07-13T02:09:57.898 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2023-07-13 02:09:26.408891238 +0000 2023-07-13T02:09:57.898 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2023-07-13 02:09:26.332892780 +0000 2023-07-13T02:09:57.898 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2023-07-13 02:09:26.332892780 +0000 2023-07-13T02:09:57.898 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2023-07-13T02:09:57.898 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2023-07-13T02:09:57.970 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2023-07-13T02:09:57.970 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2023-07-13T02:09:57.970 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000148199 s, 3.5 MB/s 2023-07-13T02:09:57.970 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2023-07-13T02:09:58.032 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/nvme2n1 2023-07-13T02:09:58.092 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/nvme2n1 2023-07-13T02:09:58.093 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-07-13T02:09:58.093 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 421260 Links: 1 Device type: 103,2 2023-07-13T02:09:58.093 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-07-13T02:09:58.093 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:nvme_device_t:s0 2023-07-13T02:09:58.093 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2023-07-13 02:09:26.417891056 +0000 2023-07-13T02:09:58.093 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2023-07-13 02:09:26.334892739 +0000 2023-07-13T02:09:58.093 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2023-07-13 02:09:26.334892739 +0000 2023-07-13T02:09:58.094 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2023-07-13T02:09:58.094 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2023-07-13T02:09:58.163 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2023-07-13T02:09:58.163 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2023-07-13T02:09:58.163 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000134855 s, 3.8 MB/s 2023-07-13T02:09:58.164 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2023-07-13T02:09:58.225 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/nvme3n1 2023-07-13T02:09:58.250 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:58 smithi191 ceph-mon[108588]: Reconfiguring mon.b (monmap changed)... 2023-07-13T02:09:58.250 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:58 smithi191 ceph-mon[108588]: Reconfiguring daemon mon.b on smithi191 2023-07-13T02:09:58.250 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:58 smithi191 ceph-mon[108588]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:58.250 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:58 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:58.251 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:58 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:58.251 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:58 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:58.251 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:58 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:58.251 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:58 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:09:58.251 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:58 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:58.251 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:58 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:58.251 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:58 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:58.251 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:58 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:58.252 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:58 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:09:58.252 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:58 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:58.252 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:58 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-13T02:09:58.252 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:58 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-13T02:09:58.252 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:58 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-13T02:09:58.252 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:58 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:58.285 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/nvme3n1 2023-07-13T02:09:58.285 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-07-13T02:09:58.286 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 421342 Links: 1 Device type: 103,3 2023-07-13T02:09:58.286 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-07-13T02:09:58.286 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:nvme_device_t:s0 2023-07-13T02:09:58.286 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2023-07-13 02:09:26.427890853 +0000 2023-07-13T02:09:58.286 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2023-07-13 02:09:26.337892678 +0000 2023-07-13T02:09:58.286 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2023-07-13 02:09:26.337892678 +0000 2023-07-13T02:09:58.286 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2023-07-13T02:09:58.287 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2023-07-13T02:09:58.352 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[116810]: Reconfiguring mon.b (monmap changed)... 2023-07-13T02:09:58.353 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[116810]: Reconfiguring daemon mon.b on smithi191 2023-07-13T02:09:58.353 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[116810]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:58.353 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:58.353 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:58.353 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:58.353 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:58.354 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:09:58.354 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:58.354 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:58.354 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:58.354 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:58.354 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:09:58.354 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:58.354 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-13T02:09:58.355 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-13T02:09:58.355 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-13T02:09:58.355 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:58.355 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[104859]: Reconfiguring mon.b (monmap changed)... 2023-07-13T02:09:58.355 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[104859]: Reconfiguring daemon mon.b on smithi191 2023-07-13T02:09:58.356 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[104859]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:09:58.356 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:58.356 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:58.356 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:58.356 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:58.356 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:09:58.356 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:58.357 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:58.357 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:09:58.357 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:58.357 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:09:58.357 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:58.357 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-13T02:09:58.357 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-13T02:09:58.357 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-13T02:09:58.358 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:58 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:09:58.358 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2023-07-13T02:09:58.359 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2023-07-13T02:09:58.359 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000131957 s, 3.9 MB/s 2023-07-13T02:09:58.359 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2023-07-13T02:09:58.420 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/nvme4n1 2023-07-13T02:09:58.479 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/nvme4n1 2023-07-13T02:09:58.480 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-07-13T02:09:58.480 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 419326 Links: 1 Device type: 103,4 2023-07-13T02:09:58.480 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-07-13T02:09:58.480 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:nvme_device_t:s0 2023-07-13T02:09:58.480 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2023-07-13 02:09:26.436890670 +0000 2023-07-13T02:09:58.480 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2023-07-13 02:09:26.339892638 +0000 2023-07-13T02:09:58.480 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2023-07-13 02:09:26.339892638 +0000 2023-07-13T02:09:58.481 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2023-07-13T02:09:58.481 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2023-07-13T02:09:58.547 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2023-07-13T02:09:58.547 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2023-07-13T02:09:58.547 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000130265 s, 3.9 MB/s 2023-07-13T02:09:58.548 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2023-07-13T02:09:58.591 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:09:58 smithi191 systemd[1]: Starting Ceph mgr.x for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:09:58.608 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-07-13T02:09:58.608 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2023-07-13T02:09:58.629 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2023-07-13T02:09:58.629 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/nvme1n1 2023-07-13T02:09:58.688 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/nvme1n1 2023-07-13T02:09:58.688 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-07-13T02:09:58.688 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 421461 Links: 1 Device type: 103,1 2023-07-13T02:09:58.688 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-07-13T02:09:58.688 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:nvme_device_t:s0 2023-07-13T02:09:58.688 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-07-13 02:09:34.662997069 +0000 2023-07-13T02:09:58.689 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-07-13 02:09:34.593998241 +0000 2023-07-13T02:09:58.689 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-07-13 02:09:34.593998241 +0000 2023-07-13T02:09:58.689 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-07-13T02:09:58.689 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2023-07-13T02:09:58.758 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-07-13T02:09:58.759 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-07-13T02:09:58.759 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.00024593 s, 2.1 MB/s 2023-07-13T02:09:58.760 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2023-07-13T02:09:58.819 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/nvme2n1 2023-07-13T02:09:58.877 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/nvme2n1 2023-07-13T02:09:58.878 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-07-13T02:09:58.878 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 422955 Links: 1 Device type: 103,2 2023-07-13T02:09:58.878 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-07-13T02:09:58.878 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:nvme_device_t:s0 2023-07-13T02:09:58.879 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-07-13 02:09:34.672996899 +0000 2023-07-13T02:09:58.879 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-07-13 02:09:34.595998207 +0000 2023-07-13T02:09:58.879 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-07-13 02:09:34.595998207 +0000 2023-07-13T02:09:58.880 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-07-13T02:09:58.880 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2023-07-13T02:09:58.945 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-07-13T02:09:58.945 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-07-13T02:09:58.946 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000218153 s, 2.3 MB/s 2023-07-13T02:09:58.947 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2023-07-13T02:09:59.008 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/nvme3n1 2023-07-13T02:09:59.066 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/nvme3n1 2023-07-13T02:09:59.067 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-07-13T02:09:59.067 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 423057 Links: 1 Device type: 103,3 2023-07-13T02:09:59.067 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-07-13T02:09:59.067 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:nvme_device_t:s0 2023-07-13T02:09:59.067 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-07-13 02:09:34.682996730 +0000 2023-07-13T02:09:59.068 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-07-13 02:09:34.598998156 +0000 2023-07-13T02:09:59.068 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-07-13 02:09:34.598998156 +0000 2023-07-13T02:09:59.068 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-07-13T02:09:59.068 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2023-07-13T02:09:59.134 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-07-13T02:09:59.134 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-07-13T02:09:59.134 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000254804 s, 2.0 MB/s 2023-07-13T02:09:59.136 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2023-07-13T02:09:59.195 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/nvme4n1 2023-07-13T02:09:59.230 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:09:58 smithi191 ceph-mgr[110538]: -- 172.21.15.191:0/1497895925 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a17a9d7520 con 0x55a17a9ad000 2023-07-13T02:09:59.230 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:09:58 smithi191 bash[110438]: 6aa8322cc77928ab81e674252c37f58e818f5cfc04afbbfb588df44ad239bd5f 2023-07-13T02:09:59.230 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:09:58 smithi191 systemd[1]: Started Ceph mgr.x for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:09:59.230 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:09:59 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:09:59.082+0000 7fbad7630200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-13T02:09:59.237 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:59 smithi191 ceph-mon[108588]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi032=y;smithi191=x", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:59.238 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:59 smithi191 ceph-mon[108588]: Saving service mgr spec with placement smithi032=y;smithi191=x;count:2 2023-07-13T02:09:59.238 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:59 smithi191 ceph-mon[108588]: Deploying daemon mgr.x on smithi191 2023-07-13T02:09:59.238 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:59 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:59.238 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:59 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:59.238 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:59 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:59.238 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:09:59 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:59.263 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/nvme4n1 2023-07-13T02:09:59.264 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-07-13T02:09:59.264 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 422171 Links: 1 Device type: 103,4 2023-07-13T02:09:59.264 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-07-13T02:09:59.264 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:nvme_device_t:s0 2023-07-13T02:09:59.264 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-07-13 02:09:34.692996560 +0000 2023-07-13T02:09:59.264 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-07-13 02:09:34.600998122 +0000 2023-07-13T02:09:59.264 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-07-13 02:09:34.600998122 +0000 2023-07-13T02:09:59.265 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-07-13T02:09:59.265 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2023-07-13T02:09:59.329 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-07-13T02:09:59.329 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-07-13T02:09:59.329 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000248192 s, 2.1 MB/s 2023-07-13T02:09:59.330 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2023-07-13T02:09:59.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:59 smithi032 ceph-mon[104859]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi032=y;smithi191=x", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:59.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:59 smithi032 ceph-mon[104859]: Saving service mgr spec with placement smithi032=y;smithi191=x;count:2 2023-07-13T02:09:59.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:59 smithi032 ceph-mon[104859]: Deploying daemon mgr.x on smithi191 2023-07-13T02:09:59.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:59 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:59.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:59 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:59.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:59 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:59.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:09:59 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:59.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:59 smithi032 ceph-mon[116810]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi032=y;smithi191=x", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:09:59.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:59 smithi032 ceph-mon[116810]: Saving service mgr spec with placement smithi032=y;smithi191=x;count:2 2023-07-13T02:09:59.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:59 smithi032 ceph-mon[116810]: Deploying daemon mgr.x on smithi191 2023-07-13T02:09:59.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:59 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:59.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:59 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:59.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:59 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:59.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:09:59 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:09:59.390 INFO:tasks.cephadm:Deploying osd.0 on smithi032 with /dev/nvme4n1... 2023-07-13T02:09:59.390 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/nvme4n1 2023-07-13T02:09:59.630 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:09:59.635 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:09:59 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:09:59.260+0000 7fbad7630200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-13T02:10:00.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:00 smithi032 ceph-mon[104859]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:00.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:00 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:00.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:00 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:00.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:00 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:00.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:00 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:00.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:00 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:00.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:00 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:00.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:00 smithi032 ceph-mon[104859]: overall HEALTH_OK 2023-07-13T02:10:00.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:00 smithi032 ceph-mon[116810]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:00.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:00 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:00.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:00 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:00.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:00 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:00.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:00 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:00.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:00 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:00.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:00 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:00.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:00 smithi032 ceph-mon[116810]: overall HEALTH_OK 2023-07-13T02:10:00.394 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:00 smithi191 ceph-mon[108588]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:00.395 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:00 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:00.395 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:00 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:00.395 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:00 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:00.395 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:00 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:00.395 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:00 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:00.395 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:00 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:00.395 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:00 smithi191 ceph-mon[108588]: overall HEALTH_OK 2023-07-13T02:10:00.396 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:09:59 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:09:59.632+0000 7fbad7630200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-13T02:10:00.396 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:10:00 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:10:00.167+0000 7fbad7630200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-13T02:10:00.396 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:10:00 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:10:00.391+0000 7fbad7630200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-13T02:10:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/nvme4n1 2023-07-13T02:10:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2023-07-13T02:10:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2023-07-13T02:10:00.852 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2023-07-13T02:10:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2023-07-13T02:10:00.852 INFO:teuthology.orchestra.run.smithi032.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011009 s, 952 MB/s 2023-07-13T02:10:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2023-07-13T02:10:01.078 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:10:00 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:10:00.804+0000 7fbad7630200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-13T02:10:01.234 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:01 smithi032 ceph-mon[116810]: Reconfiguring mgr.y (unknown last config time)... 2023-07-13T02:10:01.235 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:01 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-13T02:10:01.235 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:01 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-13T02:10:01.235 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:01 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:01.235 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:01 smithi032 ceph-mon[116810]: Reconfiguring daemon mgr.y on smithi032 2023-07-13T02:10:01.236 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:01 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:01.236 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:01 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:01.236 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:01 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:01.236 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:01 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:01.237 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:01 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:01.237 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:01 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:01.237 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:01 smithi032 ceph-mon[104859]: Reconfiguring mgr.y (unknown last config time)... 2023-07-13T02:10:01.238 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:01 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-13T02:10:01.238 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:01 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-13T02:10:01.238 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:01 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:01.238 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:01 smithi032 ceph-mon[104859]: Reconfiguring daemon mgr.y on smithi032 2023-07-13T02:10:01.238 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:01 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:01.239 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:01 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:01.239 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:01 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:01.239 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:01 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:01.239 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:01 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:01.240 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:01 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:01.295 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi032:/dev/nvme4n1 2023-07-13T02:10:01.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:01 smithi191 ceph-mon[108588]: Reconfiguring mgr.y (unknown last config time)... 2023-07-13T02:10:01.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:01 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-13T02:10:01.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:01 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-13T02:10:01.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:01 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:01.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:01 smithi191 ceph-mon[108588]: Reconfiguring daemon mgr.y on smithi032 2023-07-13T02:10:01.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:01 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:01.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:01 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:01.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:01 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:01.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:01 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:01.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:01 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:01.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:01 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:01.526 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:10:01.980 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:10:01 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:10:01.509+0000 7fbad7630200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-13T02:10:01.980 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:10:01 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:10:01.594+0000 7fbad7630200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-13T02:10:02.283 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:02 smithi032 ceph-mon[104859]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:02.283 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:02 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:02.284 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:02 smithi032 ceph-mon[116810]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:02.284 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:02 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:02.364 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:02 smithi191 ceph-mon[108588]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:02.364 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:02 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:02.364 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:10:02 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:10:02.190+0000 7fbad7630200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-13T02:10:02.364 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:10:02 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:10:02.273+0000 7fbad7630200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-13T02:10:02.681 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:10:02 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:10:02.360+0000 7fbad7630200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-13T02:10:02.682 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:10:02 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:10:02.452+0000 7fbad7630200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-13T02:10:02.980 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:10:02 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:10:02.678+0000 7fbad7630200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-13T02:10:02.980 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:10:02 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:10:02.770+0000 7fbad7630200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-13T02:10:03.265 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:10:02 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:10:02.981+0000 7fbad7630200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-13T02:10:03.265 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:10:03 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:10:03.076+0000 7fbad7630200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-13T02:10:03.266 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:10:03 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:10:03.261+0000 7fbad7630200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-13T02:10:03.266 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:03 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:10:03.266 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:03 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:10:03.267 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:03 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:03.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:03 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:10:03.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:03 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:10:03.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:03 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:03.387 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:03 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:10:03.387 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:03 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:10:03.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:03 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:03.730 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:10:03 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:10:03.347+0000 7fbad7630200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-13T02:10:04.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:04 smithi032 ceph-mon[104859]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:10:04.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:04 smithi032 ceph-mon[104859]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:04.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:04 smithi032 ceph-mon[116810]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:10:04.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:04 smithi032 ceph-mon[116810]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:04.476 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:04 smithi191 ceph-mon[108588]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:10:04.476 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:04 smithi191 ceph-mon[108588]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:04.477 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:10:04 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:10:04.104+0000 7fbad7630200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-13T02:10:04.477 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:10:04 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:10:04.296+0000 7fbad7630200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-13T02:10:05.387 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:05 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/443097682' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "daf065be-4354-4ddb-8bcb-543eeba9ebc0"}]: dispatch 2023-07-13T02:10:05.387 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:05 smithi032 ceph-mon[116810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "daf065be-4354-4ddb-8bcb-543eeba9ebc0"}]: dispatch 2023-07-13T02:10:05.387 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:05 smithi032 ceph-mon[116810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "daf065be-4354-4ddb-8bcb-543eeba9ebc0"}]': finished 2023-07-13T02:10:05.387 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:05 smithi032 ceph-mon[116810]: osdmap e5: 1 total, 0 up, 1 in 2023-07-13T02:10:05.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:05 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:10:05.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:05 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/443097682' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "daf065be-4354-4ddb-8bcb-543eeba9ebc0"}]: dispatch 2023-07-13T02:10:05.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:05 smithi032 ceph-mon[104859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "daf065be-4354-4ddb-8bcb-543eeba9ebc0"}]: dispatch 2023-07-13T02:10:05.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:05 smithi032 ceph-mon[104859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "daf065be-4354-4ddb-8bcb-543eeba9ebc0"}]': finished 2023-07-13T02:10:05.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:05 smithi032 ceph-mon[104859]: osdmap e5: 1 total, 0 up, 1 in 2023-07-13T02:10:05.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:05 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:10:05.466 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:10:04 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:10:04.472+0000 7fbad7630200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-13T02:10:05.466 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:10:05 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:10:05.103+0000 7fbad7630200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-13T02:10:05.466 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:10:05 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:10:05.282+0000 7fbad7630200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-13T02:10:05.467 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:10:05 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:10:05.462+0000 7fbad7630200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-13T02:10:05.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:05 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/443097682' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "daf065be-4354-4ddb-8bcb-543eeba9ebc0"}]: dispatch 2023-07-13T02:10:05.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:05 smithi191 ceph-mon[108588]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "daf065be-4354-4ddb-8bcb-543eeba9ebc0"}]: dispatch 2023-07-13T02:10:05.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:05 smithi191 ceph-mon[108588]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "daf065be-4354-4ddb-8bcb-543eeba9ebc0"}]': finished 2023-07-13T02:10:05.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:05 smithi191 ceph-mon[108588]: osdmap e5: 1 total, 0 up, 1 in 2023-07-13T02:10:05.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:05 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:10:05.730 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:10:05 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x[110503]: 2023-07-13T02:10:05.576+0000 7fbad7630200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-13T02:10:06.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:06 smithi032 ceph-mon[104859]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:06.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:06 smithi032 ceph-mon[104859]: Standby manager daemon x started 2023-07-13T02:10:06.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:06 smithi032 ceph-mon[104859]: from='mgr.? 172.21.15.191:0/1578432851' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-13T02:10:06.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:06 smithi032 ceph-mon[104859]: from='mgr.? 172.21.15.191:0/1578432851' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-13T02:10:06.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:06 smithi032 ceph-mon[104859]: from='mgr.? 172.21.15.191:0/1578432851' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-13T02:10:06.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:06 smithi032 ceph-mon[104859]: from='mgr.? 172.21.15.191:0/1578432851' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-13T02:10:06.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:06 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/298539746' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:10:06.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:06 smithi032 ceph-mon[116810]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:06.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:06 smithi032 ceph-mon[116810]: Standby manager daemon x started 2023-07-13T02:10:06.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:06 smithi032 ceph-mon[116810]: from='mgr.? 172.21.15.191:0/1578432851' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-13T02:10:06.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:06 smithi032 ceph-mon[116810]: from='mgr.? 172.21.15.191:0/1578432851' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-13T02:10:06.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:06 smithi032 ceph-mon[116810]: from='mgr.? 172.21.15.191:0/1578432851' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-13T02:10:06.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:06 smithi032 ceph-mon[116810]: from='mgr.? 172.21.15.191:0/1578432851' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-13T02:10:06.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:06 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/298539746' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:10:06.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:06 smithi191 ceph-mon[108588]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:06.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:06 smithi191 ceph-mon[108588]: Standby manager daemon x started 2023-07-13T02:10:06.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:06 smithi191 ceph-mon[108588]: from='mgr.? 172.21.15.191:0/1578432851' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-07-13T02:10:06.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:06 smithi191 ceph-mon[108588]: from='mgr.? 172.21.15.191:0/1578432851' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-13T02:10:06.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:06 smithi191 ceph-mon[108588]: from='mgr.? 172.21.15.191:0/1578432851' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-07-13T02:10:06.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:06 smithi191 ceph-mon[108588]: from='mgr.? 172.21.15.191:0/1578432851' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-13T02:10:06.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:06 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/298539746' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:10:07.730 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:07 smithi191 ceph-mon[108588]: mgrmap e14: y(active, since 49s), standbys: x 2023-07-13T02:10:07.731 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:07 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-13T02:10:07.731 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:07 smithi191 ceph-mon[108588]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:07.887 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:07 smithi032 ceph-mon[116810]: mgrmap e14: y(active, since 49s), standbys: x 2023-07-13T02:10:07.888 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:07 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-13T02:10:07.888 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:07 smithi032 ceph-mon[116810]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:07.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:07 smithi032 ceph-mon[104859]: mgrmap e14: y(active, since 49s), standbys: x 2023-07-13T02:10:07.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:07 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-13T02:10:07.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:07 smithi032 ceph-mon[104859]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:10.084 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:10 smithi032 ceph-mon[104859]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:10.084 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:10 smithi032 ceph-mon[116810]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:10.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:10 smithi191 ceph-mon[108588]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:11.289 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:11 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-13T02:10:11.289 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:11 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:11.289 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:11 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-13T02:10:11.290 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:11 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:11.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:11 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-13T02:10:11.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:11 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:12.200 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:12 smithi032 ceph-mon[104859]: Deploying daemon osd.0 on smithi032 2023-07-13T02:10:12.201 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:12 smithi032 ceph-mon[104859]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:12.201 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:12 smithi032 ceph-mon[116810]: Deploying daemon osd.0 on smithi032 2023-07-13T02:10:12.201 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:12 smithi032 ceph-mon[116810]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:12.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:12 smithi191 ceph-mon[108588]: Deploying daemon osd.0 on smithi032 2023-07-13T02:10:12.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:12 smithi191 ceph-mon[108588]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:14.177 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:14 smithi032 ceph-mon[104859]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:14.178 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:14 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:14.178 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:14 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:14.178 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:14 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:14.178 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:14 smithi032 ceph-mon[116810]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:14.178 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:14 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:14.178 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:14 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:14.179 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:14 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:14.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:14 smithi191 ceph-mon[108588]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:14.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:14 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:14.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:14 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:14.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:14 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:16.152 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 0 on host 'smithi032' 2023-07-13T02:10:16.321 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:16 smithi032 ceph-mon[104859]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:16.321 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:16 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:16.321 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:16 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:16.322 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:16 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:16.322 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:16 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:16.322 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:16 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:16.322 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:16 smithi032 ceph-mon[116810]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:16.322 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:16 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:16.322 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:16 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:16.323 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:16 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:16.323 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:16 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:16.323 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:16 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:16.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:16 smithi191 ceph-mon[108588]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:16.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:16 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:16.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:16 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:16.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:16 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:16.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:16 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:16.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:16 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:16.532 DEBUG:teuthology.orchestra.run.smithi032:osd.0> sudo journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service 2023-07-13T02:10:16.535 INFO:tasks.cephadm:Deploying osd.1 on smithi032 with /dev/nvme3n1... 2023-07-13T02:10:16.535 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/nvme3n1 2023-07-13T02:10:16.568 INFO:journalctl@ceph.osd.0.smithi032.stdout:-- Logs begin at Thu 2023-07-13 01:52:05 UTC. -- 2023-07-13T02:10:17.106 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:10:17.293 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:17 smithi032 ceph-mon[104859]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:17.294 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:17 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:17.294 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:17 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:17.294 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:17 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:17.294 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:17 smithi032 ceph-mon[104859]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:10:17.295 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:17 smithi032 ceph-mon[116810]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:17.295 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:17 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:17.295 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:17 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:17.295 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:17 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:17.295 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:17 smithi032 ceph-mon[116810]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:10:17.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:17 smithi191 ceph-mon[108588]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:17.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:17 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:17.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:17 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:17.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:17 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:17.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:17 smithi191 ceph-mon[108588]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:10:18.505 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/nvme3n1 2023-07-13T02:10:18.505 INFO:teuthology.orchestra.run.smithi032.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2023-07-13T02:10:18.506 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme3n1 bs=1M count=10 conv=fsync 2023-07-13T02:10:18.506 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2023-07-13T02:10:18.506 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2023-07-13T02:10:18.506 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0117754 s, 890 MB/s 2023-07-13T02:10:18.506 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2023-07-13T02:10:18.917 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi032:/dev/nvme3n1 2023-07-13T02:10:19.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:19 smithi032 ceph-mon[104859]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:19.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:19 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:19.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:19 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:19.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:19 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:10:19.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:19 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:19.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:19 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:19.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:19 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:19.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:19 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:19.139 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:19 smithi032 ceph-mon[116810]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:19.139 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:19 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:19.139 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:19 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:19.139 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:19 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:10:19.140 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:19 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:19.140 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:19 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:19.140 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:19 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:19.140 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:19 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:19.149 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:10:19.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:19 smithi191 ceph-mon[108588]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:19.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:19 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:19.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:19 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:19.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:19 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:10:19.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:19 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:19.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:19 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:19.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:19 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:19.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:19 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:20.072 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:20 smithi032 ceph-mon[116810]: Detected new or changed devices on smithi032 2023-07-13T02:10:20.073 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:20 smithi032 ceph-mon[116810]: Adjusting osd_memory_target on smithi032 to 16102M 2023-07-13T02:10:20.073 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:20 smithi032 ceph-mon[116810]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:20.073 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:20 smithi032 ceph-mon[116810]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:20.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:20 smithi032 ceph-mon[104859]: Detected new or changed devices on smithi032 2023-07-13T02:10:20.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:20 smithi032 ceph-mon[104859]: Adjusting osd_memory_target on smithi032 to 16102M 2023-07-13T02:10:20.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:20 smithi032 ceph-mon[104859]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:20.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:20 smithi032 ceph-mon[104859]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:20.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:20 smithi191 ceph-mon[108588]: Detected new or changed devices on smithi032 2023-07-13T02:10:20.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:20 smithi191 ceph-mon[108588]: Adjusting osd_memory_target on smithi032 to 16102M 2023-07-13T02:10:20.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:20 smithi191 ceph-mon[108588]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:20.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:20 smithi191 ceph-mon[108588]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:21.321 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:21 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:10:21.321 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:21 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:10:21.321 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:21 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:21.322 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:21 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:10:21.322 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:21 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:10:21.322 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:21 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:21.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:21 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:10:21.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:21 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:10:21.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:21 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:22.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:22 smithi032 ceph-mon[104859]: from='client.24134 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:10:22.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:22 smithi032 ceph-mon[104859]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:22.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:22 smithi032 ceph-mon[116810]: from='client.24134 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:10:22.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:22 smithi032 ceph-mon[116810]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:22.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:22 smithi191 ceph-mon[108588]: from='client.24134 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:10:22.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:22 smithi191 ceph-mon[108588]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:23.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:23 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1736173647' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bceb0f23-2c64-4f43-a428-bdd76b1a2368"}]: dispatch 2023-07-13T02:10:23.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:23 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1736173647' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bceb0f23-2c64-4f43-a428-bdd76b1a2368"}]': finished 2023-07-13T02:10:23.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:23 smithi032 ceph-mon[104859]: osdmap e6: 2 total, 0 up, 2 in 2023-07-13T02:10:23.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:23 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:10:23.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:23 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-13T02:10:23.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:23 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1736173647' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bceb0f23-2c64-4f43-a428-bdd76b1a2368"}]: dispatch 2023-07-13T02:10:23.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:23 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1736173647' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bceb0f23-2c64-4f43-a428-bdd76b1a2368"}]': finished 2023-07-13T02:10:23.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:23 smithi032 ceph-mon[116810]: osdmap e6: 2 total, 0 up, 2 in 2023-07-13T02:10:23.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:23 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:10:23.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:23 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-13T02:10:23.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:23 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1736173647' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bceb0f23-2c64-4f43-a428-bdd76b1a2368"}]: dispatch 2023-07-13T02:10:23.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:23 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1736173647' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bceb0f23-2c64-4f43-a428-bdd76b1a2368"}]': finished 2023-07-13T02:10:23.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:23 smithi191 ceph-mon[108588]: osdmap e6: 2 total, 0 up, 2 in 2023-07-13T02:10:23.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:23 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:10:23.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:23 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-13T02:10:24.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:24 smithi032 ceph-mon[104859]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:24.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:24 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2810226537' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:10:24.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:24 smithi032 ceph-mon[116810]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:24.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:24 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2810226537' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:10:24.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:24 smithi191 ceph-mon[108588]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:24.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:24 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/2810226537' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:10:26.008 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:25 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:10:26.008 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:25 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Scheduled restart job, restart counter is at 1. 2023-07-13T02:10:26.008 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:25 smithi032 systemd[1]: Stopped Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:10:26.008 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:25 smithi032 systemd[1]: Starting Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:10:26.282 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:26 smithi032 ceph-mon[104859]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:26.282 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:26 smithi032 ceph-mon[116810]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:26.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:26 smithi191 ceph-mon[108588]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:26.887 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:26 smithi032 bash[126181]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:10:26.888 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:26 smithi032 bash[126181]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:10:26.888 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:26 smithi032 bash[126181]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:10:26.888 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:26 smithi032 bash[126181]: --> Failed to activate any OSD(s) 2023-07-13T02:10:27.971 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:27 smithi032 ceph-osd[126395]: -- 172.21.15.32:0/1034766676 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556b2da6ed00 con 0x556b2da3ec00 2023-07-13T02:10:27.971 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:27 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-0[126387]: 2023-07-13T02:10:27.710+0000 7f10e18a2640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:10:27.971 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:27 smithi032 bash[126371]: 24870071d8627fc2979482de91298e184c25ec0ca93d313613c6ce84f2a79bd0 2023-07-13T02:10:27.971 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:27 smithi032 systemd[1]: Started Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:10:27.971 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:27 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:10:28.323 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:28 smithi032 ceph-mon[104859]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:28.323 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:28 smithi032 ceph-mon[116810]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:28.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:28 smithi191 ceph-mon[108588]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:29.387 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:29 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-13T02:10:29.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:29 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:29.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:29 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-13T02:10:29.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:29 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:29.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:29 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-13T02:10:29.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:29 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:29.797 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:29 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Failed with result 'exit-code'. 2023-07-13T02:10:30.117 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:30 smithi032 ceph-mon[104859]: Deploying daemon osd.1 on smithi032 2023-07-13T02:10:30.117 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:30 smithi032 ceph-mon[104859]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:30.387 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:30 smithi032 ceph-mon[116810]: Deploying daemon osd.1 on smithi032 2023-07-13T02:10:30.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:30 smithi032 ceph-mon[116810]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:30.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:30 smithi191 ceph-mon[108588]: Deploying daemon osd.1 on smithi032 2023-07-13T02:10:30.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:30 smithi191 ceph-mon[108588]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:31.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:31 smithi032 ceph-mon[104859]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:31.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:31 smithi032 ceph-mon[116810]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:31.480 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:31 smithi191 ceph-mon[108588]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:33.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:33 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:33.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:33 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:33.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:33 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:33.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:33 smithi032 ceph-mon[104859]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:33.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:33 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:33.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:33 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:33.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:33 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:33.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:33 smithi032 ceph-mon[116810]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:33.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:33 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:33.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:33 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:33.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:33 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:33.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:33 smithi191 ceph-mon[108588]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:34.794 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 1 on host 'smithi032' 2023-07-13T02:10:35.170 DEBUG:teuthology.orchestra.run.smithi032:osd.1> sudo journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service 2023-07-13T02:10:35.172 INFO:tasks.cephadm:Deploying osd.2 on smithi032 with /dev/nvme2n1... 2023-07-13T02:10:35.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/nvme2n1 2023-07-13T02:10:35.205 INFO:journalctl@ceph.osd.1.smithi032.stdout:-- Logs begin at Thu 2023-07-13 01:52:05 UTC. -- 2023-07-13T02:10:35.637 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:35 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:35.638 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:35 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:35.638 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:35 smithi032 ceph-mon[104859]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:35.638 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:35 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:35.638 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:35 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:35.638 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:35 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:35.639 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:35 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:35.639 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:35 smithi032 ceph-mon[116810]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:35.639 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:35 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:35.639 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:35 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:35.837 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:10:35.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:35 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:35.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:35 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:35.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:35 smithi191 ceph-mon[108588]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:35.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:35 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:35.981 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:35 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:37.088 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/nvme2n1 2023-07-13T02:10:37.088 INFO:teuthology.orchestra.run.smithi032.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2023-07-13T02:10:37.089 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme2n1 bs=1M count=10 conv=fsync 2023-07-13T02:10:37.089 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2023-07-13T02:10:37.089 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2023-07-13T02:10:37.089 INFO:teuthology.orchestra.run.smithi032.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0135588 s, 773 MB/s 2023-07-13T02:10:37.089 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2023-07-13T02:10:37.558 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi032:/dev/nvme2n1 2023-07-13T02:10:37.799 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:10:37.868 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:37 smithi032 ceph-mon[104859]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:37.868 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:37 smithi032 ceph-mon[116810]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:37.980 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:37 smithi191 ceph-mon[108588]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:39.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[104859]: Detected new or changed devices on smithi032 2023-07-13T02:10:39.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:39.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:39.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:10:39.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:10:39.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[104859]: Adjusting osd_memory_target on smithi032 to 8051M 2023-07-13T02:10:39.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:39.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:39.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:39.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[104859]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:39.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[104859]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:39.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:39.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:39.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:39.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:39.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:39.140 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[116810]: Detected new or changed devices on smithi032 2023-07-13T02:10:39.140 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:39.141 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:39.141 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:10:39.141 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:10:39.141 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[116810]: Adjusting osd_memory_target on smithi032 to 8051M 2023-07-13T02:10:39.141 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:39.141 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:39.141 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:39.141 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[116810]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:39.142 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[116810]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:39.142 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:39.142 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:39.142 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:39.142 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:39.142 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:38 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:39.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:38 smithi191 ceph-mon[108588]: Detected new or changed devices on smithi032 2023-07-13T02:10:39.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:38 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:39.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:38 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:39.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:38 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:10:39.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:38 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:10:39.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:38 smithi191 ceph-mon[108588]: Adjusting osd_memory_target on smithi032 to 8051M 2023-07-13T02:10:39.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:38 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:39.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:38 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:39.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:38 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:39.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:38 smithi191 ceph-mon[108588]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:39.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:38 smithi191 ceph-mon[108588]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:39.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:38 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:39.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:38 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:39.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:38 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:39.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:38 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:39.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:38 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:39.798 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:39 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:10:39.798 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:39 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Scheduled restart job, restart counter is at 2. 2023-07-13T02:10:39.799 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:39 smithi032 systemd[1]: Stopped Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:10:39.799 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:39 smithi032 systemd[1]: Starting Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:10:40.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:39 smithi032 ceph-mon[104859]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:40.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:39 smithi032 ceph-mon[104859]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:40.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:39 smithi032 ceph-mon[104859]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:10:40.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:39 smithi032 ceph-mon[104859]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:10:40.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:39 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:10:40.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:39 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:10:40.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:39 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:40.139 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:39 smithi032 ceph-mon[116810]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:40.139 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:39 smithi032 ceph-mon[116810]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:40.140 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:39 smithi032 ceph-mon[116810]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:10:40.140 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:39 smithi032 ceph-mon[116810]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:10:40.140 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:39 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:10:40.140 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:39 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:10:40.140 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:39 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:40.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:39 smithi191 ceph-mon[108588]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:40.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:39 smithi191 ceph-mon[108588]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:40.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:39 smithi191 ceph-mon[108588]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:10:40.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:39 smithi191 ceph-mon[108588]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:10:40.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:39 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:10:40.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:39 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:10:40.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:39 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:40.888 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:40 smithi032 bash[131157]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:10:40.888 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:40 smithi032 bash[131157]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:10:40.888 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:40 smithi032 bash[131157]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:10:40.888 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:40 smithi032 bash[131157]: --> Failed to activate any OSD(s) 2023-07-13T02:10:41.158 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:41 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3403085127' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81e90675-502b-4b7f-b41a-98bc60ac1821"}]: dispatch 2023-07-13T02:10:41.158 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:41 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3403085127' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "81e90675-502b-4b7f-b41a-98bc60ac1821"}]': finished 2023-07-13T02:10:41.158 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:41 smithi032 ceph-mon[104859]: osdmap e7: 3 total, 0 up, 3 in 2023-07-13T02:10:41.158 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:41 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:10:41.158 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:41 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-13T02:10:41.159 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:41 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-13T02:10:41.159 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:41 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3403085127' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81e90675-502b-4b7f-b41a-98bc60ac1821"}]: dispatch 2023-07-13T02:10:41.159 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:41 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3403085127' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "81e90675-502b-4b7f-b41a-98bc60ac1821"}]': finished 2023-07-13T02:10:41.159 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:41 smithi032 ceph-mon[116810]: osdmap e7: 3 total, 0 up, 3 in 2023-07-13T02:10:41.159 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:41 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:10:41.159 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:41 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-13T02:10:41.160 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:41 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-13T02:10:41.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:41 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3403085127' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81e90675-502b-4b7f-b41a-98bc60ac1821"}]: dispatch 2023-07-13T02:10:41.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:41 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3403085127' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "81e90675-502b-4b7f-b41a-98bc60ac1821"}]': finished 2023-07-13T02:10:41.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:41 smithi191 ceph-mon[108588]: osdmap e7: 3 total, 0 up, 3 in 2023-07-13T02:10:41.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:41 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:10:41.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:41 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-13T02:10:41.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:41 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-13T02:10:41.638 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:41 smithi032 ceph-osd[131337]: -- 172.21.15.32:0/2481852658 <== mon.2 v2:172.21.15.32:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fa999cad00 con 0x55fa9999ac00 2023-07-13T02:10:41.638 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:41 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-0[131328]: 2023-07-13T02:10:41.250+0000 7fbd52e8e640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:10:41.638 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:41 smithi032 bash[131303]: 5fe1b8b0accf03123ff9bc6e08b68e38bf5d7ce253d55a5f311e5162d64efde0 2023-07-13T02:10:41.638 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:41 smithi032 systemd[1]: Started Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:10:42.074 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:41 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:10:42.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:42 smithi032 ceph-mon[116810]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:42.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:42 smithi032 ceph-mon[104859]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:42.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:42 smithi191 ceph-mon[108588]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:43.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:43 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/74262260' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:10:43.138 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:43 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/74262260' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:10:43.139 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:42 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Failed with result 'exit-code'. 2023-07-13T02:10:43.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:43 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/74262260' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:10:44.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:44 smithi032 ceph-mon[104859]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:44.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:44 smithi032 ceph-mon[116810]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:44.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:44 smithi191 ceph-mon[108588]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:44.702 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:44 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:10:44.703 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:44 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Scheduled restart job, restart counter is at 1. 2023-07-13T02:10:44.703 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:44 smithi032 systemd[1]: Stopped Ceph osd.1 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:10:44.703 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:44 smithi032 systemd[1]: Starting Ceph osd.1 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:10:45.387 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:45 smithi032 bash[132514]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:10:45.388 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:45 smithi032 bash[132514]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:10:45.388 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:45 smithi032 bash[132514]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:10:45.388 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:45 smithi032 bash[132514]: --> Failed to activate any OSD(s) 2023-07-13T02:10:46.127 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:46 smithi032 ceph-mon[116810]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:46.127 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:46 smithi032 ceph-mon[104859]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:46.127 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:45 smithi032 ceph-osd[132783]: -- 172.21.15.32:0/330467115 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55febeab0d00 con 0x55febea80c00 2023-07-13T02:10:46.128 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:45 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-1[132764]: 2023-07-13T02:10:45.867+0000 7f8f6c9cc640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:10:46.128 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:45 smithi032 bash[132745]: 954663db9ef764f77eddce21d184490849a31e49381720eeef25b7d032ecdf97 2023-07-13T02:10:46.128 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:45 smithi032 systemd[1]: Started Ceph osd.1 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:10:46.128 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:46 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:10:46.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:46 smithi191 ceph-mon[108588]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:48.084 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:47 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Failed with result 'exit-code'. 2023-07-13T02:10:48.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:48 smithi032 ceph-mon[104859]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:48.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:48 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-13T02:10:48.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:48 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:48.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:48 smithi032 ceph-mon[116810]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:48.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:48 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-13T02:10:48.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:48 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:48.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:48 smithi191 ceph-mon[108588]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:48.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:48 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-13T02:10:48.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:48 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:49.354 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:49 smithi032 ceph-mon[116810]: Deploying daemon osd.2 on smithi032 2023-07-13T02:10:49.355 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:49 smithi032 ceph-mon[104859]: Deploying daemon osd.2 on smithi032 2023-07-13T02:10:49.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:49 smithi191 ceph-mon[108588]: Deploying daemon osd.2 on smithi032 2023-07-13T02:10:50.335 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:50 smithi032 ceph-mon[104859]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:50.335 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:50 smithi032 ceph-mon[116810]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:50.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:50 smithi191 ceph-mon[108588]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:52.138 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:51 smithi032 ceph-mon[116810]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:52.139 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:51 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:52.139 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:51 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:52.139 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:51 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:52.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:51 smithi032 ceph-mon[104859]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:52.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:51 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:52.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:51 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:52.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:51 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:52.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:51 smithi191 ceph-mon[108588]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:52.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:51 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:52.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:51 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:52.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:51 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:53.159 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:52 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:10:53.160 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:52 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Scheduled restart job, restart counter is at 3. 2023-07-13T02:10:53.160 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:52 smithi032 systemd[1]: Stopped Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:10:53.160 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:52 smithi032 systemd[1]: Starting Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:10:53.365 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 2 on host 'smithi032' 2023-07-13T02:10:54.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:53 smithi032 ceph-mon[104859]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:54.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:53 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:54.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:53 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:54.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:53 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:54.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:53 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:54.140 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:53 smithi032 ceph-mon[116810]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:54.140 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:53 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:54.141 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:53 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:54.141 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:53 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:54.141 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:53 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:54.142 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:53 smithi032 bash[134130]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:10:54.142 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:53 smithi032 bash[134130]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:10:54.142 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:53 smithi032 bash[134130]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:10:54.142 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:53 smithi032 bash[134130]: --> Failed to activate any OSD(s) 2023-07-13T02:10:54.148 DEBUG:teuthology.orchestra.run.smithi032:osd.2> sudo journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service 2023-07-13T02:10:54.150 INFO:tasks.cephadm:Deploying osd.3 on smithi032 with /dev/nvme1n1... 2023-07-13T02:10:54.151 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/nvme1n1 2023-07-13T02:10:54.182 INFO:journalctl@ceph.osd.2.smithi032.stdout:-- Logs begin at Thu 2023-07-13 01:52:05 UTC. -- 2023-07-13T02:10:54.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:53 smithi191 ceph-mon[108588]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:54.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:53 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:54.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:53 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:54.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:53 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:54.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:53 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:54.610 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:10:55.276 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:55 smithi032 ceph-osd[134552]: -- 172.21.15.32:0/2787138167 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56343974cd00 con 0x56343971cc00 2023-07-13T02:10:55.276 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:55 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-0[134531]: 2023-07-13T02:10:55.012+0000 7f3558553640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:10:55.277 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:55 smithi032 bash[134499]: cf493f9b71487df2f57e3554ec16a1fdc219cd340e91177d07cca5cecd318e80 2023-07-13T02:10:55.277 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:55 smithi032 systemd[1]: Started Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:10:55.630 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:55 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:10:56.215 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:56 smithi032 ceph-mon[104859]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:56.216 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:56 smithi032 ceph-mon[116810]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:56.402 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/nvme1n1 2023-07-13T02:10:56.402 INFO:teuthology.orchestra.run.smithi032.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2023-07-13T02:10:56.402 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme1n1 bs=1M count=10 conv=fsync 2023-07-13T02:10:56.402 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2023-07-13T02:10:56.402 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2023-07-13T02:10:56.403 INFO:teuthology.orchestra.run.smithi032.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.010976 s, 955 MB/s 2023-07-13T02:10:56.403 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2023-07-13T02:10:56.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:56 smithi191 ceph-mon[108588]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:56.960 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi032:/dev/nvme1n1 2023-07-13T02:10:57.212 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:10:57.326 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:10:57 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Failed with result 'exit-code'. 2023-07-13T02:10:57.991 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:57 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:10:57.991 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:57 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Scheduled restart job, restart counter is at 2. 2023-07-13T02:10:57.991 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:57 smithi032 systemd[1]: Stopped Ceph osd.1 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:10:57.992 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:57 smithi032 systemd[1]: Starting Ceph osd.1 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:10:57.992 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[104859]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:57.992 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[104859]: Detected new or changed devices on smithi032 2023-07-13T02:10:58.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:58.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:58.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:10:58.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:10:58.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:10:58.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[104859]: Adjusting osd_memory_target on smithi032 to 5367M 2023-07-13T02:10:58.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:58.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:58.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:58.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[104859]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:58.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[104859]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:58.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:58.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[104859]: Health check update: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:10:58.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:58.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:58.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:58.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:58.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[116810]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:58.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[116810]: Detected new or changed devices on smithi032 2023-07-13T02:10:58.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:58.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:58.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:10:58.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:10:58.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:10:58.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[116810]: Adjusting osd_memory_target on smithi032 to 5367M 2023-07-13T02:10:58.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:58.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:58.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:58.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[116810]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:58.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[116810]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:58.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:58.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[116810]: Health check update: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:10:58.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:58.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:58.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:58.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:57 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:58.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:57 smithi191 ceph-mon[108588]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:58.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:57 smithi191 ceph-mon[108588]: Detected new or changed devices on smithi032 2023-07-13T02:10:58.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:57 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:58.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:57 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:58.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:57 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:10:58.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:57 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:10:58.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:57 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:10:58.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:57 smithi191 ceph-mon[108588]: Adjusting osd_memory_target on smithi032 to 5367M 2023-07-13T02:10:58.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:57 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:58.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:57 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:58.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:57 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:58.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:57 smithi191 ceph-mon[108588]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:58.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:57 smithi191 ceph-mon[108588]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:58.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:57 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:58.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:57 smithi191 ceph-mon[108588]: Health check update: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:10:58.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:57 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:10:58.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:57 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:58.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:57 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:10:58.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:57 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:10:58.647 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:58 smithi032 bash[137616]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:10:58.648 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:58 smithi032 bash[137616]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:10:58.648 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:58 smithi032 bash[137616]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:10:58.648 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:58 smithi032 bash[137616]: --> Failed to activate any OSD(s) 2023-07-13T02:10:59.253 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:59 smithi032 ceph-mon[116810]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:59.253 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:59 smithi032 ceph-mon[116810]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:59.253 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:59 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:10:59.254 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:59 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:10:59.254 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:10:59 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:59.254 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:59 smithi032 ceph-mon[104859]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:59.254 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:59 smithi032 ceph-mon[104859]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:59.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:59 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:10:59.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:59 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:10:59.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:10:59 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:59.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:59 smithi191 ceph-mon[108588]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:59.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:59 smithi191 ceph-mon[108588]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:10:59.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:59 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:10:59.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:59 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:10:59.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:10:59 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:10:59.568 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:59 smithi032 bash[137859]: 8a34f53a490a11d13d3302ea9f53e288493eb5631d66f346197d748376feb65e 2023-07-13T02:10:59.569 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:59 smithi032 ceph-osd[137916]: -- 172.21.15.32:0/665165292 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557eb876cd00 con 0x557eb873d400 2023-07-13T02:10:59.569 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:59 smithi032 systemd[1]: Started Ceph osd.1 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:10:59.569 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:59 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-1[137912]: 2023-07-13T02:10:59.479+0000 7f01ae9f0640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:11:00.331 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:00 smithi032 ceph-mon[104859]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:11:00.332 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:00 smithi032 ceph-mon[116810]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:11:00.332 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:10:59 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:11:00.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:00 smithi191 ceph-mon[108588]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:11:01.389 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:01 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Failed with result 'exit-code'. 2023-07-13T02:11:01.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:01 smithi032 ceph-mon[104859]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:01.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:01 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/904391599' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91bbdd2f-4a5a-4103-8a9d-2f7684d3250e"}]: dispatch 2023-07-13T02:11:01.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:01 smithi032 ceph-mon[104859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91bbdd2f-4a5a-4103-8a9d-2f7684d3250e"}]: dispatch 2023-07-13T02:11:01.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:01 smithi032 ceph-mon[104859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91bbdd2f-4a5a-4103-8a9d-2f7684d3250e"}]': finished 2023-07-13T02:11:01.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:01 smithi032 ceph-mon[104859]: osdmap e8: 4 total, 0 up, 4 in 2023-07-13T02:11:01.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:01 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:11:01.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:01 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-13T02:11:01.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:01 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-13T02:11:01.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:01 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-13T02:11:01.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:01 smithi032 ceph-mon[116810]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:01.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:01 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/904391599' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91bbdd2f-4a5a-4103-8a9d-2f7684d3250e"}]: dispatch 2023-07-13T02:11:01.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:01 smithi032 ceph-mon[116810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91bbdd2f-4a5a-4103-8a9d-2f7684d3250e"}]: dispatch 2023-07-13T02:11:01.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:01 smithi032 ceph-mon[116810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91bbdd2f-4a5a-4103-8a9d-2f7684d3250e"}]': finished 2023-07-13T02:11:01.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:01 smithi032 ceph-mon[116810]: osdmap e8: 4 total, 0 up, 4 in 2023-07-13T02:11:01.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:01 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:11:01.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:01 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-13T02:11:01.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:01 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-13T02:11:01.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:01 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-13T02:11:01.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:01 smithi191 ceph-mon[108588]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:01.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:01 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/904391599' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91bbdd2f-4a5a-4103-8a9d-2f7684d3250e"}]: dispatch 2023-07-13T02:11:01.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:01 smithi191 ceph-mon[108588]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91bbdd2f-4a5a-4103-8a9d-2f7684d3250e"}]: dispatch 2023-07-13T02:11:01.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:01 smithi191 ceph-mon[108588]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91bbdd2f-4a5a-4103-8a9d-2f7684d3250e"}]': finished 2023-07-13T02:11:01.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:01 smithi191 ceph-mon[108588]: osdmap e8: 4 total, 0 up, 4 in 2023-07-13T02:11:01.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:01 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:11:01.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:01 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-13T02:11:01.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:01 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-13T02:11:01.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:01 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-13T02:11:02.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:02 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/932870385' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:11:02.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:02 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/932870385' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:11:02.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:02 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/932870385' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:11:03.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:03 smithi032 ceph-mon[104859]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:03.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:03 smithi032 ceph-mon[116810]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:03.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:03 smithi191 ceph-mon[108588]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:04.031 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:03 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:11:04.031 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:03 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Scheduled restart job, restart counter is at 1. 2023-07-13T02:11:04.031 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:03 smithi032 systemd[1]: Stopped Ceph osd.2 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:04.031 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:03 smithi032 systemd[1]: Starting Ceph osd.2 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:11:04.638 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:04 smithi032 bash[139023]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:04.638 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:04 smithi032 bash[139023]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:04.638 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:04 smithi032 bash[139023]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:11:04.638 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:04 smithi032 bash[139023]: --> Failed to activate any OSD(s) 2023-07-13T02:11:05.370 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:05 smithi032 ceph-mon[116810]: pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:05.370 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:05 smithi032 ceph-mon[104859]: pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:05.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:05 smithi191 ceph-mon[108588]: pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:05.989 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:05 smithi032 ceph-osd[139469]: -- 172.21.15.32:0/1553545072 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56332767ad00 con 0x56332764b400 2023-07-13T02:11:05.989 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:05 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-2[139458]: 2023-07-13T02:11:05.671+0000 7facf2301640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:11:05.989 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:05 smithi032 bash[139423]: d23290e97f67750fdf13bf3bef42d7f838ba744cf1a0a7f8e3962320e771d9f9 2023-07-13T02:11:05.989 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:05 smithi032 systemd[1]: Started Ceph osd.2 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:06.291 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:06 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:11:07.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:07 smithi032 ceph-mon[104859]: pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:07.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:07 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-13T02:11:07.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:07 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:07.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:07 smithi032 ceph-mon[116810]: pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:07.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:07 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-13T02:11:07.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:07 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:07.391 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:07 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:11:07.391 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:07 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Scheduled restart job, restart counter is at 4. 2023-07-13T02:11:07.392 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:07 smithi032 systemd[1]: Stopped Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:07.392 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:07 smithi032 systemd[1]: Starting Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:11:07.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:07 smithi191 ceph-mon[108588]: pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:07.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:07 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-13T02:11:07.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:07 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:07.776 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:07 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Failed with result 'exit-code'. 2023-07-13T02:11:08.366 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:08 smithi032 ceph-mon[116810]: Deploying daemon osd.3 on smithi032 2023-07-13T02:11:08.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:08 smithi032 ceph-mon[104859]: Deploying daemon osd.3 on smithi032 2023-07-13T02:11:08.367 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:08 smithi032 bash[139834]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:08.367 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:08 smithi032 bash[139834]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:08.367 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:08 smithi032 bash[139834]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:11:08.367 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:08 smithi032 bash[139834]: --> Failed to activate any OSD(s) 2023-07-13T02:11:08.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:08 smithi191 ceph-mon[108588]: Deploying daemon osd.3 on smithi032 2023-07-13T02:11:09.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:09 smithi032 ceph-mon[104859]: pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:09.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:09 smithi032 ceph-mon[116810]: pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:09.389 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:08 smithi032 bash[139986]: ff6d4bb1f52379e99e98816e26d5c3786115eb4f1d8a83107587bfc9f2647c16 2023-07-13T02:11:09.389 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:08 smithi032 systemd[1]: Started Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:09.390 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:08 smithi032 ceph-osd[140040]: -- 172.21.15.32:0/802372895 <== mon.2 v2:172.21.15.32:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55876df88d00 con 0x55876df58c00 2023-07-13T02:11:09.390 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:08 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-0[140036]: 2023-07-13T02:11:08.977+0000 7fe10d88c640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:11:09.390 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:09 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:11:09.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:09 smithi191 ceph-mon[108588]: pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:10.639 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:10 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Failed with result 'exit-code'. 2023-07-13T02:11:11.293 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:11 smithi032 ceph-mon[104859]: pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:11.293 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:11 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:11.293 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:11 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:11.294 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:11 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:11:11.294 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:11 smithi032 ceph-mon[116810]: pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:11.294 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:11 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:11.294 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:11 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:11.295 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:11 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:11:11.295 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:11 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:11:11.295 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:11 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Scheduled restart job, restart counter is at 3. 2023-07-13T02:11:11.295 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:11 smithi032 systemd[1]: Stopped Ceph osd.1 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:11.295 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:11 smithi032 systemd[1]: Starting Ceph osd.1 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:11:11.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:11 smithi191 ceph-mon[108588]: pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:11.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:11 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:11.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:11 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:11.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:11 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:11:12.139 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:11 smithi032 bash[140727]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:12.139 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:11 smithi032 bash[140727]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:12.139 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:11 smithi032 bash[140727]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:11:12.139 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:11 smithi032 bash[140727]: --> Failed to activate any OSD(s) 2023-07-13T02:11:13.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:13 smithi032 ceph-mon[104859]: pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:13.367 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:13 smithi032 ceph-mon[116810]: pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:13.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:13 smithi191 ceph-mon[108588]: pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:13.888 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:13 smithi032 ceph-osd[141132]: -- 172.21.15.32:0/3898164934 <== mon.2 v2:172.21.15.32:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557be23e2d00 con 0x557be23b2c00 2023-07-13T02:11:13.889 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:13 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-1[141108]: 2023-07-13T02:11:13.428+0000 7f4f61967640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:11:13.889 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:13 smithi032 bash[141022]: af9b106db78ad1284448f349d782653e6600d95c64da1f0b6a057de7afeb6238 2023-07-13T02:11:13.889 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:13 smithi032 systemd[1]: Started Ceph osd.1 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:14.252 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:13 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:11:14.293 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 3 on host 'smithi032' 2023-07-13T02:11:14.718 DEBUG:teuthology.orchestra.run.smithi032:osd.3> sudo journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.3.service 2023-07-13T02:11:14.720 INFO:tasks.cephadm:Deploying osd.4 on smithi191 with /dev/nvme4n1... 2023-07-13T02:11:14.720 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/nvme4n1 2023-07-13T02:11:14.758 INFO:journalctl@ceph.osd.3.smithi032.stdout:-- Logs begin at Thu 2023-07-13 01:52:05 UTC. -- 2023-07-13T02:11:14.953 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.b/config 2023-07-13T02:11:15.207 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:15 smithi032 ceph-mon[116810]: pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:15.207 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:15 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:15.208 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:15 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:15.208 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:15 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:15.208 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:15 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:15.208 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:15 smithi032 ceph-mon[104859]: pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:15.209 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:15 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:15.209 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:15 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:15.209 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:15 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:15.209 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:15 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:15.404 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:15 smithi191 ceph-mon[108588]: pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:15.404 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:15 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:15.405 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:15 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:15.405 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:15 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:15.405 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:15 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:15.933 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:15 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Failed with result 'exit-code'. 2023-07-13T02:11:16.110 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/nvme4n1 2023-07-13T02:11:16.111 INFO:teuthology.orchestra.run.smithi191.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2023-07-13T02:11:16.111 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2023-07-13T02:11:16.111 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2023-07-13T02:11:16.111 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2023-07-13T02:11:16.111 INFO:teuthology.orchestra.run.smithi191.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0139975 s, 749 MB/s 2023-07-13T02:11:16.112 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2023-07-13T02:11:16.487 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi191:/dev/nvme4n1 2023-07-13T02:11:16.708 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.b/config 2023-07-13T02:11:17.185 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:17 smithi032 ceph-mon[116810]: pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:17.185 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:17 smithi032 ceph-mon[104859]: pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:17.377 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:17 smithi191 ceph-mon[108588]: pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:17.893 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:17 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:11:17.894 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:17 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Scheduled restart job, restart counter is at 2. 2023-07-13T02:11:17.894 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:17 smithi032 systemd[1]: Stopped Ceph osd.2 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:17.894 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:17 smithi032 systemd[1]: Starting Ceph osd.2 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:11:18.454 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:18.454 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: Detected new or changed devices on smithi032 2023-07-13T02:11:18.454 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:18.455 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:18.455 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:11:18.455 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:11:18.455 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:11:18.455 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:11:18.455 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: Adjusting osd_memory_target on smithi032 to 4025M 2023-07-13T02:11:18.455 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:18.456 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:18.456 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:11:18.456 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:18.456 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:18.456 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:18.456 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:11:18.456 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:18.457 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:11:18.457 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:18.457 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:18.457 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:18.457 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:11:18.457 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:11:18.457 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:11:18.457 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:18.458 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:18 smithi191 ceph-mon[108588]: Health check update: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:11:18.638 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:18.639 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: Detected new or changed devices on smithi032 2023-07-13T02:11:18.639 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:18.639 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:18.639 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:11:18.639 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:11:18.639 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:11:18.640 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:11:18.640 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: Adjusting osd_memory_target on smithi032 to 4025M 2023-07-13T02:11:18.640 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:18.640 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:18.640 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:11:18.640 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:18.640 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:18.641 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:18.641 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:11:18.641 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:18.641 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:11:18.641 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:18.641 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:18.641 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:18.641 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:11:18.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:11:18.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:11:18.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:18.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[104859]: Health check update: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:11:18.642 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:18.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: Detected new or changed devices on smithi032 2023-07-13T02:11:18.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:18.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:18.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:11:18.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:11:18.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:11:18.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:11:18.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: Adjusting osd_memory_target on smithi032 to 4025M 2023-07-13T02:11:18.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:18.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:18.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:11:18.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:18.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:18.645 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:18.645 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:11:18.645 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:18.645 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:11:18.645 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:18.645 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:18.645 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:18.645 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:11:18.646 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:11:18.646 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:11:18.646 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:18.646 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-mon[116810]: Health check update: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:11:18.646 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:18 smithi032 bash[144409]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:18.646 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:18 smithi032 bash[144409]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:18.647 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:18 smithi032 bash[144409]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:11:18.647 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:18 smithi032 bash[144409]: --> Failed to activate any OSD(s) 2023-07-13T02:11:19.199 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-osd[144515]: -- 172.21.15.32:0/275553174 <== mon.2 v2:172.21.15.32:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c34f030d00 con 0x55c34f001000 2023-07-13T02:11:19.199 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:18 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-2[144511]: 2023-07-13T02:11:18.888+0000 7f80af5f4640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:11:19.199 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:18 smithi032 bash[144496]: dfca963cfec819d90c93f30c35d0cdf7f5d3422eddb231613b3335936e78285e 2023-07-13T02:11:19.199 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:18 smithi032 systemd[1]: Started Ceph osd.2 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:19.485 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:19 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:11:19.486 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:19 smithi032 ceph-mon[104859]: from='client.? 172.21.15.191:0/153842753' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d84e61b6-aee0-4a38-bca6-29128f793567"}]: dispatch 2023-07-13T02:11:19.486 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:19 smithi032 ceph-mon[104859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d84e61b6-aee0-4a38-bca6-29128f793567"}]: dispatch 2023-07-13T02:11:19.486 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:19 smithi032 ceph-mon[104859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d84e61b6-aee0-4a38-bca6-29128f793567"}]': finished 2023-07-13T02:11:19.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:19 smithi032 ceph-mon[104859]: osdmap e9: 5 total, 0 up, 5 in 2023-07-13T02:11:19.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:19 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:11:19.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:19 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-13T02:11:19.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:19 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-13T02:11:19.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:19 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-13T02:11:19.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:19 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-13T02:11:19.890 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:19 smithi032 ceph-mon[116810]: from='client.? 172.21.15.191:0/153842753' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d84e61b6-aee0-4a38-bca6-29128f793567"}]: dispatch 2023-07-13T02:11:19.890 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:19 smithi032 ceph-mon[116810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d84e61b6-aee0-4a38-bca6-29128f793567"}]: dispatch 2023-07-13T02:11:19.890 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:19 smithi032 ceph-mon[116810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d84e61b6-aee0-4a38-bca6-29128f793567"}]': finished 2023-07-13T02:11:19.890 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:19 smithi032 ceph-mon[116810]: osdmap e9: 5 total, 0 up, 5 in 2023-07-13T02:11:19.890 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:19 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:11:19.890 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:19 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-13T02:11:19.890 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:19 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-13T02:11:19.891 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:19 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-13T02:11:19.891 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:19 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-13T02:11:19.982 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:19 smithi191 ceph-mon[108588]: from='client.? 172.21.15.191:0/153842753' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d84e61b6-aee0-4a38-bca6-29128f793567"}]: dispatch 2023-07-13T02:11:19.983 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:19 smithi191 ceph-mon[108588]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d84e61b6-aee0-4a38-bca6-29128f793567"}]: dispatch 2023-07-13T02:11:19.983 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:19 smithi191 ceph-mon[108588]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d84e61b6-aee0-4a38-bca6-29128f793567"}]': finished 2023-07-13T02:11:19.983 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:19 smithi191 ceph-mon[108588]: osdmap e9: 5 total, 0 up, 5 in 2023-07-13T02:11:19.983 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:19 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:11:19.983 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:19 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-13T02:11:19.983 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:19 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-13T02:11:19.983 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:19 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-13T02:11:19.984 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:19 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-13T02:11:20.639 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:20 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:11:20.639 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:20 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Failed with result 'exit-code'. 2023-07-13T02:11:20.911 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:20 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Scheduled restart job, restart counter is at 5. 2023-07-13T02:11:20.912 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:20 smithi032 systemd[1]: Stopped Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:20.912 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:20 smithi032 systemd[1]: Starting Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:11:21.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:21 smithi032 ceph-mon[104859]: pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:21.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:21 smithi032 ceph-mon[104859]: from='client.? 172.21.15.191:0/2691353571' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:11:21.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:21 smithi032 ceph-mon[116810]: pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:21.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:21 smithi032 ceph-mon[116810]: from='client.? 172.21.15.191:0/2691353571' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:11:21.391 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:21 smithi032 bash[144711]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:21.391 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:21 smithi032 bash[144711]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:21.391 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:21 smithi032 bash[144711]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:11:21.391 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:21 smithi032 bash[144711]: --> Failed to activate any OSD(s) 2023-07-13T02:11:21.481 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:21 smithi191 ceph-mon[108588]: pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:21.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:21 smithi191 ceph-mon[108588]: from='client.? 172.21.15.191:0/2691353571' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:11:22.048 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:21 smithi032 ceph-osd[144813]: -- 172.21.15.32:0/160589233 <== mon.2 v2:172.21.15.32:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b6672fcd00 con 0x55b6672cd400 2023-07-13T02:11:22.048 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:21 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-0[144809]: 2023-07-13T02:11:21.789+0000 7fa965429640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:11:22.049 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:21 smithi032 bash[144795]: 4ece3c391e0ad5fec76a2a0b14f0569c36cd4407e01d3938fde021f2f66d3741 2023-07-13T02:11:22.049 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:21 smithi032 systemd[1]: Started Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:22.050 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:22 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:11:23.138 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:23 smithi032 ceph-mon[116810]: pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:23.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:23 smithi032 ceph-mon[104859]: pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:23.311 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:23 smithi191 ceph-mon[108588]: pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:23.639 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:23 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Failed with result 'exit-code'. 2023-07-13T02:11:24.408 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:24 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.3.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:11:24.408 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:24 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.3.service: Scheduled restart job, restart counter is at 1. 2023-07-13T02:11:24.409 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:24 smithi032 systemd[1]: Stopped Ceph osd.3 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:24.409 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:24 smithi032 systemd[1]: Starting Ceph osd.3 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:11:25.077 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:24 smithi032 bash[145007]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:25.077 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:24 smithi032 bash[145007]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:25.077 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:24 smithi032 bash[145007]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:11:25.078 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:24 smithi032 bash[145007]: --> Failed to activate any OSD(s) 2023-07-13T02:11:25.335 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:25 smithi032 ceph-mon[116810]: pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:25.335 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:25 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-13T02:11:25.335 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:25 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:25.336 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:25 smithi032 ceph-mon[104859]: pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:25.336 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:25 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-13T02:11:25.336 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:25 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:25.414 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:25 smithi191 ceph-mon[108588]: pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:25.414 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:25 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-13T02:11:25.414 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:25 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:25.597 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:25 smithi032 ceph-osd[145110]: -- 172.21.15.32:0/3845894328 <== mon.2 v2:172.21.15.32:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559dc8aced00 con 0x559dc8a9f400 2023-07-13T02:11:25.598 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:25 smithi032 bash[145092]: 41dd2456a8772d40d2653c698cc477ea3f2ea52a96cf024d446debc2119d6cb5 2023-07-13T02:11:25.598 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:25 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-3[145106]: 2023-07-13T02:11:25.341+0000 7f48605f7640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:11:25.598 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:25 smithi032 systemd[1]: Started Ceph osd.3 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:26.091 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:26 smithi191 ceph-mon[108588]: Deploying daemon osd.4 on smithi191 2023-07-13T02:11:26.264 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:26 smithi032 ceph-mon[116810]: Deploying daemon osd.4 on smithi191 2023-07-13T02:11:26.265 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:25 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.3.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:11:26.266 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:26 smithi032 ceph-mon[104859]: Deploying daemon osd.4 on smithi191 2023-07-13T02:11:26.266 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:25 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:11:26.267 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:25 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Scheduled restart job, restart counter is at 4. 2023-07-13T02:11:26.267 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:25 smithi032 systemd[1]: Stopped Ceph osd.1 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:26.267 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:25 smithi032 systemd[1]: Starting Ceph osd.1 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:11:26.889 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:26 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.3.service: Failed with result 'exit-code'. 2023-07-13T02:11:26.889 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:26 smithi032 bash[145283]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:26.889 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:26 smithi032 bash[145283]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:26.889 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:26 smithi032 bash[145283]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:11:26.890 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:26 smithi032 bash[145283]: --> Failed to activate any OSD(s) 2023-07-13T02:11:27.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:27 smithi191 ceph-mon[108588]: pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:27.335 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:27 smithi032 ceph-mon[116810]: pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:27.336 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:27 smithi032 ceph-mon[104859]: pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:27.336 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:27 smithi032 ceph-osd[145410]: -- 172.21.15.32:0/1782961623 <== mon.2 v2:172.21.15.32:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555d6c70ed00 con 0x555d6c6df000 2023-07-13T02:11:27.336 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:27 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-1[145406]: 2023-07-13T02:11:27.317+0000 7fa3b5891640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:11:27.336 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:27 smithi032 bash[145391]: a796e50517e3c6fa3f12a8c8227e5e246a00f1503bdbdaf754a7bb0efd4477cd 2023-07-13T02:11:27.336 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:27 smithi032 systemd[1]: Started Ceph osd.1 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:27.888 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:27 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:11:29.086 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:28 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Failed with result 'exit-code'. 2023-07-13T02:11:29.087 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:29 smithi032 ceph-mon[104859]: pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:29.087 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:29 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:29.087 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:29 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:29.087 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:29 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:11:29.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:29 smithi191 ceph-mon[108588]: pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:29.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:29 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:29.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:29 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:29.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:29 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:11:29.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:29 smithi032 ceph-mon[116810]: pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:29.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:29 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:29.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:29 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:29.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:29 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:11:30.354 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 4 on host 'smithi191' 2023-07-13T02:11:30.644 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:30 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:11:30.645 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:30 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Scheduled restart job, restart counter is at 3. 2023-07-13T02:11:30.645 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:30 smithi032 systemd[1]: Stopped Ceph osd.2 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:30.645 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:30 smithi032 systemd[1]: Starting Ceph osd.2 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:11:30.733 DEBUG:teuthology.orchestra.run.smithi191:osd.4> sudo journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service 2023-07-13T02:11:30.735 INFO:tasks.cephadm:Deploying osd.5 on smithi191 with /dev/nvme3n1... 2023-07-13T02:11:30.735 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/nvme3n1 2023-07-13T02:11:30.768 INFO:journalctl@ceph.osd.4.smithi191.stdout:-- Logs begin at Thu 2023-07-13 01:53:22 UTC. -- 2023-07-13T02:11:31.227 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:31 smithi191 ceph-mon[108588]: pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:31.227 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:31 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:31.227 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:31 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:31.227 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:31 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:31.227 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:31 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:11:31.228 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:31 smithi191 ceph-mon[108588]: pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:31.228 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:31 smithi191 ceph-mon[108588]: pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:31.228 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:31 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:31.228 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:31 smithi191 ceph-mon[108588]: Health check update: 5 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:11:31.228 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:31 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:31.228 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:31 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:31.228 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:31 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:11:31.239 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.b/config 2023-07-13T02:11:31.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[104859]: pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:31.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:31.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:31.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:31.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:11:31.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[104859]: pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:31.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[104859]: pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:31.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:31.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[104859]: Health check update: 5 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:11:31.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:31.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:31.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:11:31.392 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:30 smithi032 bash[145643]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:31.393 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:30 smithi032 bash[145643]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:31.393 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:30 smithi032 bash[145643]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:11:31.393 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:30 smithi032 bash[145643]: --> Failed to activate any OSD(s) 2023-07-13T02:11:31.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[116810]: pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:31.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:31.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:31.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:31.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:11:31.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[116810]: pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:31.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[116810]: pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:31.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:31.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[116810]: Health check update: 5 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:11:31.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:31.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:31.395 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:11:31.889 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-osd[145746]: -- 172.21.15.32:0/3567823924 <== mon.2 v2:172.21.15.32:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5577d7188d00 con 0x5577d7159000 2023-07-13T02:11:31.889 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:31 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-2[145742]: 2023-07-13T02:11:31.647+0000 7f0eb7e43640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:11:31.889 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:31 smithi032 bash[145728]: 5e1bbe0f64f39a77585d6330ff5eb2e169d79e4d3db4fd2744788a4822e3d4d2 2023-07-13T02:11:31.889 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:31 smithi032 systemd[1]: Started Ceph osd.2 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:32.330 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:31 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:11:32.549 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/nvme3n1 2023-07-13T02:11:32.550 INFO:teuthology.orchestra.run.smithi191.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2023-07-13T02:11:32.550 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme3n1 bs=1M count=10 conv=fsync 2023-07-13T02:11:32.550 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2023-07-13T02:11:32.550 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2023-07-13T02:11:32.550 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0144569 s, 725 MB/s 2023-07-13T02:11:32.551 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2023-07-13T02:11:33.027 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi191:/dev/nvme3n1 2023-07-13T02:11:33.249 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.b/config 2023-07-13T02:11:33.347 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:33 smithi191 ceph-mon[108588]: pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:33.348 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:33 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:33.348 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:33 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:33.348 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:33 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:11:33.348 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:33 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:33.348 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:33 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:33.349 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:33 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:11:33.349 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:33 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:33.388 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:33 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Failed with result 'exit-code'. 2023-07-13T02:11:33.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:33 smithi032 ceph-mon[104859]: pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:33.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:33 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:33.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:33 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:33.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:33 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:11:33.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:33 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:33.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:33 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:33.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:33 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:11:33.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:33 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:33.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:33 smithi032 ceph-mon[116810]: pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:33.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:33 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:33.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:33 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:33.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:33 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:11:33.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:33 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:33.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:33 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:33.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:33 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:11:33.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:33 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:33.687 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:33 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:11:33.687 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:33 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Scheduled restart job, restart counter is at 6. 2023-07-13T02:11:33.687 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:33 smithi032 systemd[1]: Stopped Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:33.688 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:33 smithi032 systemd[1]: Starting Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:11:34.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:34 smithi032 ceph-mon[104859]: Detected new or changed devices on smithi191 2023-07-13T02:11:34.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:34 smithi032 ceph-mon[104859]: Adjusting osd_memory_target on smithi191 to 17122M 2023-07-13T02:11:34.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:34 smithi032 ceph-mon[104859]: pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:34.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:34 smithi032 ceph-mon[104859]: pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:34.390 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:33 smithi032 bash[145978]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:34.391 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:33 smithi032 bash[145978]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:34.391 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:33 smithi032 bash[145978]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:11:34.391 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:33 smithi032 bash[145978]: --> Failed to activate any OSD(s) 2023-07-13T02:11:34.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:34 smithi032 ceph-mon[116810]: Detected new or changed devices on smithi191 2023-07-13T02:11:34.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:34 smithi032 ceph-mon[116810]: Adjusting osd_memory_target on smithi191 to 17122M 2023-07-13T02:11:34.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:34 smithi032 ceph-mon[116810]: pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:34.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:34 smithi032 ceph-mon[116810]: pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:34.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:34 smithi191 ceph-mon[108588]: Detected new or changed devices on smithi191 2023-07-13T02:11:34.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:34 smithi191 ceph-mon[108588]: Adjusting osd_memory_target on smithi191 to 17122M 2023-07-13T02:11:34.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:34 smithi191 ceph-mon[108588]: pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:34.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:34 smithi191 ceph-mon[108588]: pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:34.871 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:34 smithi032 ceph-osd[146082]: -- 172.21.15.32:0/4205545525 <== mon.2 v2:172.21.15.32:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cd05ac0d00 con 0x55cd05a91400 2023-07-13T02:11:34.871 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:34 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-0[146078]: 2023-07-13T02:11:34.601+0000 7f3ddc22f640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:11:34.872 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:34 smithi032 bash[146062]: e615acb8f39c1b50b4cee995e0bc52110cf92b4a86e1c1386ebc0ecf7060b89b 2023-07-13T02:11:34.872 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:34 smithi032 systemd[1]: Started Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:34.872 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:34 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:11:35.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:35 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:11:35.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:35 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:11:35.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:35 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:35.139 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:35 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:11:35.140 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:35 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:11:35.140 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:35 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:35.332 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:35 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:11:35.332 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:35 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:11:35.332 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:35 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:36.389 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:35 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Failed with result 'exit-code'. 2023-07-13T02:11:36.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[104859]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:11:36.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[104859]: pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:36.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[104859]: from='client.? 172.21.15.191:0/210459887' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58a2e747-c3ce-494c-8736-ff98ed0223ee"}]: dispatch 2023-07-13T02:11:36.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[104859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58a2e747-c3ce-494c-8736-ff98ed0223ee"}]: dispatch 2023-07-13T02:11:36.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[104859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58a2e747-c3ce-494c-8736-ff98ed0223ee"}]': finished 2023-07-13T02:11:36.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[104859]: osdmap e10: 6 total, 0 up, 6 in 2023-07-13T02:11:36.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:11:36.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-13T02:11:36.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-13T02:11:36.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-13T02:11:36.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-13T02:11:36.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-13T02:11:36.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[116810]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:11:36.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[116810]: pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:36.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[116810]: from='client.? 172.21.15.191:0/210459887' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58a2e747-c3ce-494c-8736-ff98ed0223ee"}]: dispatch 2023-07-13T02:11:36.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[116810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58a2e747-c3ce-494c-8736-ff98ed0223ee"}]: dispatch 2023-07-13T02:11:36.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[116810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58a2e747-c3ce-494c-8736-ff98ed0223ee"}]': finished 2023-07-13T02:11:36.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[116810]: osdmap e10: 6 total, 0 up, 6 in 2023-07-13T02:11:36.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:11:36.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-13T02:11:36.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-13T02:11:36.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-13T02:11:36.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-13T02:11:36.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:36 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-13T02:11:36.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:36 smithi191 ceph-mon[108588]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:11:36.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:36 smithi191 ceph-mon[108588]: pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:36.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:36 smithi191 ceph-mon[108588]: from='client.? 172.21.15.191:0/210459887' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58a2e747-c3ce-494c-8736-ff98ed0223ee"}]: dispatch 2023-07-13T02:11:36.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:36 smithi191 ceph-mon[108588]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58a2e747-c3ce-494c-8736-ff98ed0223ee"}]: dispatch 2023-07-13T02:11:36.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:36 smithi191 ceph-mon[108588]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58a2e747-c3ce-494c-8736-ff98ed0223ee"}]': finished 2023-07-13T02:11:36.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:36 smithi191 ceph-mon[108588]: osdmap e10: 6 total, 0 up, 6 in 2023-07-13T02:11:36.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:36 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:11:36.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:36 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-13T02:11:36.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:36 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-13T02:11:36.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:36 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-13T02:11:36.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:36 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-13T02:11:36.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:36 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-13T02:11:37.140 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:36 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.3.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:11:37.141 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:36 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.3.service: Scheduled restart job, restart counter is at 2. 2023-07-13T02:11:37.141 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:36 smithi032 systemd[1]: Stopped Ceph osd.3 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:37.141 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:36 smithi032 systemd[1]: Starting Ceph osd.3 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:11:37.889 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:37 smithi032 bash[146278]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:37.889 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:37 smithi032 bash[146278]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:37.889 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:37 smithi032 bash[146278]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:11:37.889 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:37 smithi032 bash[146278]: --> Failed to activate any OSD(s) 2023-07-13T02:11:38.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:38 smithi032 ceph-mon[116810]: pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:38.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:38 smithi032 ceph-mon[116810]: from='client.? 172.21.15.191:0/2994261545' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:11:38.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:38 smithi032 ceph-mon[104859]: pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:38.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:38 smithi032 ceph-mon[104859]: from='client.? 172.21.15.191:0/2994261545' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:11:38.390 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:38 smithi032 ceph-osd[146384]: -- 172.21.15.32:0/3376394388 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d0d16b8d00 con 0x55d0d1688c00 2023-07-13T02:11:38.390 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:38 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-3[146380]: 2023-07-13T02:11:38.175+0000 7fa7ce56d640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:11:38.390 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:38 smithi032 bash[146365]: d33ec80b43e40831ea0149b9c519c3f3f2ba27f1a35e7e2b6c3aface743f4daa 2023-07-13T02:11:38.390 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:38 smithi032 systemd[1]: Started Ceph osd.3 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:38.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:38 smithi191 ceph-mon[108588]: pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:38.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:38 smithi191 ceph-mon[108588]: from='client.? 172.21.15.191:0/2994261545' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:11:38.837 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:38 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.3.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:11:39.123 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:38 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:11:39.124 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:38 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Scheduled restart job, restart counter is at 5. 2023-07-13T02:11:39.124 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:38 smithi032 systemd[1]: Stopped Ceph osd.1 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:39.124 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:38 smithi032 systemd[1]: Starting Ceph osd.1 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:11:39.889 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:39 smithi032 bash[146554]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:39.890 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:39 smithi032 bash[146554]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:39.890 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:39 smithi032 bash[146554]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:11:39.890 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:39 smithi032 bash[146554]: --> Failed to activate any OSD(s) 2023-07-13T02:11:39.890 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:39 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.3.service: Failed with result 'exit-code'. 2023-07-13T02:11:40.218 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:40 smithi191 ceph-mon[108588]: pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:40.218 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:39 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:11:40.219 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:39 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Scheduled restart job, restart counter is at 1. 2023-07-13T02:11:40.219 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:39 smithi191 systemd[1]: Stopped Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:40.219 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:39 smithi191 systemd[1]: Starting Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:11:40.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:40 smithi032 ceph-mon[104859]: pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:40.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:40 smithi032 ceph-mon[116810]: pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:40.389 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:40 smithi032 ceph-osd[146681]: -- 172.21.15.32:0/2387310649 <== mon.2 v2:172.21.15.32:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5610343f2d00 con 0x5610343c2c00 2023-07-13T02:11:40.390 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:40 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-1[146677]: 2023-07-13T02:11:40.184+0000 7f2480523640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:11:40.390 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:40 smithi032 bash[146661]: f7cc6aaf7c10d62f35aae0e53331d622569f8eff2a8cc32a8032a304f607f952 2023-07-13T02:11:40.390 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:40 smithi032 systemd[1]: Started Ceph osd.1 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:40.774 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:40 smithi191 bash[118427]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:40.774 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:40 smithi191 bash[118427]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:40.775 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:40 smithi191 bash[118427]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:11:40.775 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:40 smithi191 bash[118427]: --> Failed to activate any OSD(s) 2023-07-13T02:11:40.808 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:40 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:11:41.639 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:41 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Failed with result 'exit-code'. 2023-07-13T02:11:41.982 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:41 smithi191 bash[118613]: 03c265975e4e8f13424a232a518bdbf9019aa1adc7d1ac6d8c1ab857d5bae25d 2023-07-13T02:11:41.982 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:41 smithi191 systemd[1]: Started Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:41.982 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:41 smithi191 ceph-osd[118639]: -- 172.21.15.191:0/1103288014 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ba6fe10d00 con 0x55ba6fde1000 2023-07-13T02:11:41.983 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:41 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-4[118631]: 2023-07-13T02:11:41.653+0000 7f007259b640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:11:41.983 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:41 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:11:42.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:42 smithi032 ceph-mon[104859]: pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:42.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:42 smithi032 ceph-mon[116810]: pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:42.479 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:42 smithi191 ceph-mon[108588]: pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:43.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:43 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-13T02:11:43.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:43 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:43.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:43 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-13T02:11:43.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:43 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:43.390 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:43 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:11:43.390 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:43 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Scheduled restart job, restart counter is at 4. 2023-07-13T02:11:43.390 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:43 smithi032 systemd[1]: Stopped Ceph osd.2 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:43.390 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:43 smithi032 systemd[1]: Starting Ceph osd.2 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:11:43.440 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:43 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Failed with result 'exit-code'. 2023-07-13T02:11:43.441 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:43 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-13T02:11:43.441 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:43 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:44.091 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:43 smithi032 bash[146878]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:44.092 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:43 smithi032 bash[146878]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:44.092 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:43 smithi032 bash[146878]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:11:44.092 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:43 smithi032 bash[146878]: --> Failed to activate any OSD(s) 2023-07-13T02:11:44.329 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:44 smithi191 ceph-mon[108588]: Deploying daemon osd.5 on smithi191 2023-07-13T02:11:44.329 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:44 smithi191 ceph-mon[108588]: pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:44.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:44 smithi032 ceph-mon[104859]: Deploying daemon osd.5 on smithi191 2023-07-13T02:11:44.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:44 smithi032 ceph-mon[104859]: pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:44.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:44 smithi032 ceph-mon[116810]: Deploying daemon osd.5 on smithi191 2023-07-13T02:11:44.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:44 smithi032 ceph-mon[116810]: pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:44.787 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:44 smithi032 ceph-osd[146981]: -- 172.21.15.32:0/1868477040 <== mon.2 v2:172.21.15.32:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cd382dcd00 con 0x55cd382ad000 2023-07-13T02:11:44.787 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:44 smithi032 bash[146962]: 45ab3f94f4957db6387673210e56aae879c3bd7cc3686a2d8ffda13f824a3a55 2023-07-13T02:11:44.787 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:44 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-2[146977]: 2023-07-13T02:11:44.529+0000 7ff726060640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:11:44.787 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:44 smithi032 systemd[1]: Started Ceph osd.2 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:45.537 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:44 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:11:46.139 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:46 smithi032 ceph-mon[116810]: pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:46.139 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:46 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:46.140 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:46 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:46.140 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:46 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:11:46.141 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:45 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Failed with result 'exit-code'. 2023-07-13T02:11:46.141 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:46 smithi032 ceph-mon[104859]: pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:46.142 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:46 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:46.142 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:46 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:46.142 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:46 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:11:46.394 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:46 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:11:46.395 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:46 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Scheduled restart job, restart counter is at 7. 2023-07-13T02:11:46.395 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:46 smithi032 systemd[1]: Stopped Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:46.395 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:46 smithi032 systemd[1]: Starting Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:11:46.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:46 smithi191 ceph-mon[108588]: pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:46.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:46 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:46.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:46 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:46.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:46 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:11:47.139 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:46 smithi032 bash[147287]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:47.139 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:46 smithi032 bash[147287]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:47.139 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:46 smithi032 bash[147287]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:11:47.139 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:46 smithi032 bash[147287]: --> Failed to activate any OSD(s) 2023-07-13T02:11:47.639 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:47 smithi032 ceph-osd[147392]: -- 172.21.15.32:0/4234480110 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5572504ccd00 con 0x55725049d400 2023-07-13T02:11:47.639 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:47 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-0[147388]: 2023-07-13T02:11:47.405+0000 7f0f695f5640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:11:47.639 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:47 smithi032 bash[147373]: 983bf0bca91f3860dd5fc705a5b468ee036d0be1ae8dcf9dceb36665e548333e 2023-07-13T02:11:47.640 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:47 smithi032 systemd[1]: Started Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:48.075 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:47 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:11:48.167 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 5 on host 'smithi191' 2023-07-13T02:11:48.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:48 smithi191 ceph-mon[108588]: pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:48.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:48 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:48.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:48 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:48.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:48 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:48.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:48 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:11:48.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:48 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:48.375 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:48 smithi032 ceph-mon[104859]: pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:48.375 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:48 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:48.375 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:48 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:48.375 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:48 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:48.375 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:48 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:11:48.376 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:48 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:48.376 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:48 smithi032 ceph-mon[116810]: pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:48.376 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:48 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:48.376 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:48 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:48.376 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:48 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:48.377 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:48 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:11:48.377 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:48 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:48.548 DEBUG:teuthology.orchestra.run.smithi191:osd.5> sudo journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service 2023-07-13T02:11:48.550 INFO:tasks.cephadm:Deploying osd.6 on smithi191 with /dev/nvme2n1... 2023-07-13T02:11:48.550 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/nvme2n1 2023-07-13T02:11:48.581 INFO:journalctl@ceph.osd.5.smithi191.stdout:-- Logs begin at Thu 2023-07-13 01:53:22 UTC. -- 2023-07-13T02:11:48.854 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.b/config 2023-07-13T02:11:49.124 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:48 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Failed with result 'exit-code'. 2023-07-13T02:11:49.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:49 smithi032 ceph-mon[104859]: pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:49.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:49 smithi032 ceph-mon[104859]: pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:49.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:49 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:49.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:49 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:11:49.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:49 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:49.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:49 smithi032 ceph-mon[104859]: Health check update: 6 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:11:49.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:49 smithi032 ceph-mon[116810]: pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:49.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:49 smithi032 ceph-mon[116810]: pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:49.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:49 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:49.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:49 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:11:49.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:49 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:49.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:49 smithi032 ceph-mon[116810]: Health check update: 6 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:11:49.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:49 smithi191 ceph-mon[108588]: pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:49.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:49 smithi191 ceph-mon[108588]: pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:49.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:49 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:49.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:49 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:11:49.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:49 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:49.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:49 smithi191 ceph-mon[108588]: Health check update: 6 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:11:49.899 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:49 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.3.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:11:49.899 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:49 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.3.service: Scheduled restart job, restart counter is at 3. 2023-07-13T02:11:49.900 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:49 smithi032 systemd[1]: Stopped Ceph osd.3 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:49.900 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:49 smithi032 systemd[1]: Starting Ceph osd.3 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:11:50.192 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/nvme2n1 2023-07-13T02:11:50.192 INFO:teuthology.orchestra.run.smithi191.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2023-07-13T02:11:50.192 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme2n1 bs=1M count=10 conv=fsync 2023-07-13T02:11:50.192 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2023-07-13T02:11:50.192 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2023-07-13T02:11:50.192 INFO:teuthology.orchestra.run.smithi191.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0175676 s, 597 MB/s 2023-07-13T02:11:50.193 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2023-07-13T02:11:50.208 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:50 smithi032 bash[147625]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:50.208 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:50 smithi032 bash[147625]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:50.208 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:50 smithi032 bash[147625]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:11:50.208 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:50 smithi032 bash[147625]: --> Failed to activate any OSD(s) 2023-07-13T02:11:50.616 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi191:/dev/nvme2n1 2023-07-13T02:11:50.844 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.b/config 2023-07-13T02:11:51.126 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:50 smithi032 bash[147713]: 64166f81429487574d69392d043caeaa4fbe0151b28f5cea77dff8210d51c449 2023-07-13T02:11:51.126 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:50 smithi032 ceph-osd[147731]: -- 172.21.15.32:0/206142384 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bfab7ccd00 con 0x55bfab79d000 2023-07-13T02:11:51.126 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:50 smithi032 systemd[1]: Started Ceph osd.3 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:51.126 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:50 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-3[147727]: 2023-07-13T02:11:50.855+0000 7fa0045fe640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:11:51.389 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:51 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.3.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:11:51.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:51 smithi032 ceph-mon[104859]: pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:51.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:51 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:51.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:51 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:51.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:51 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:11:51.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:51 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:11:51.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:51 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:51.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:51 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:51.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:51 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:11:51.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:51 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:51.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:51 smithi032 ceph-mon[116810]: pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:51.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:51 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:51.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:51 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:51.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:51 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:11:51.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:51 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:11:51.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:51 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:51.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:51 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:51.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:51 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:11:51.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:51 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:51.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:51 smithi191 ceph-mon[108588]: pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:51.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:51 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:51.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:51 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:51.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:51 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:11:51.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:51 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:11:51.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:51 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:51.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:51 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:51.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:51 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:11:51.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:51 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:11:51.868 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:51 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:11:51.868 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:51 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Scheduled restart job, restart counter is at 6. 2023-07-13T02:11:51.869 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:51 smithi032 systemd[1]: Stopped Ceph osd.1 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:51.869 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:51 smithi032 systemd[1]: Starting Ceph osd.1 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:11:52.389 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:11:52 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.3.service: Failed with result 'exit-code'. 2023-07-13T02:11:52.390 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:52 smithi032 bash[147936]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:52.390 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:52 smithi032 bash[147936]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:52.390 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:52 smithi032 bash[147936]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:11:52.390 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:52 smithi032 bash[147936]: --> Failed to activate any OSD(s) 2023-07-13T02:11:52.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:52 smithi032 ceph-mon[104859]: Detected new or changed devices on smithi191 2023-07-13T02:11:52.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:52 smithi032 ceph-mon[104859]: Adjusting osd_memory_target on smithi191 to 8561M 2023-07-13T02:11:52.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:52 smithi032 ceph-mon[104859]: pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:52.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:52 smithi032 ceph-mon[104859]: pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:52.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:52 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:11:52.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:52 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:11:52.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:52 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:52.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:52 smithi032 ceph-mon[116810]: Detected new or changed devices on smithi191 2023-07-13T02:11:52.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:52 smithi032 ceph-mon[116810]: Adjusting osd_memory_target on smithi191 to 8561M 2023-07-13T02:11:52.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:52 smithi032 ceph-mon[116810]: pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:52.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:52 smithi032 ceph-mon[116810]: pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:52.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:52 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:11:52.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:52 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:11:52.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:52 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:52.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:52 smithi191 ceph-mon[108588]: Detected new or changed devices on smithi191 2023-07-13T02:11:52.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:52 smithi191 ceph-mon[108588]: Adjusting osd_memory_target on smithi191 to 8561M 2023-07-13T02:11:52.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:52 smithi191 ceph-mon[108588]: pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:52.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:52 smithi191 ceph-mon[108588]: pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:52.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:52 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:11:52.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:52 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:11:52.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:52 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:11:53.132 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:52 smithi032 ceph-osd[148067]: -- 172.21.15.32:0/679235702 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563c98ac4d00 con 0x563c98a95400 2023-07-13T02:11:53.132 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:52 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-1[148063]: 2023-07-13T02:11:52.901+0000 7fb261422640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:11:53.132 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:52 smithi032 bash[148049]: 5508465f80ae6d8a36c2105664a35c281776ceb35d18d54d7b05f16850e99c48 2023-07-13T02:11:53.133 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:52 smithi032 systemd[1]: Started Ceph osd.1 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:53.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:53 smithi032 ceph-mon[104859]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:11:53.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:53 smithi032 ceph-mon[104859]: pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:53.389 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:53 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:11:53.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:53 smithi032 ceph-mon[116810]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:11:53.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:53 smithi032 ceph-mon[116810]: pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:53.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:53 smithi191 ceph-mon[108588]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:11:53.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:53 smithi191 ceph-mon[108588]: pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:53.746 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:53 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:11:53.747 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:53 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Scheduled restart job, restart counter is at 2. 2023-07-13T02:11:53.747 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:53 smithi191 systemd[1]: Stopped Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:53.747 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:53 smithi191 systemd[1]: Starting Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:11:54.270 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:54 smithi032 ceph-mon[104859]: from='client.? 172.21.15.191:0/60220453' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0386ce92-f17d-4b67-ad4e-df60b9ee4438"}]: dispatch 2023-07-13T02:11:54.270 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:54 smithi032 ceph-mon[104859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0386ce92-f17d-4b67-ad4e-df60b9ee4438"}]: dispatch 2023-07-13T02:11:54.270 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:54 smithi032 ceph-mon[104859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0386ce92-f17d-4b67-ad4e-df60b9ee4438"}]': finished 2023-07-13T02:11:54.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:54 smithi032 ceph-mon[104859]: osdmap e11: 7 total, 0 up, 7 in 2023-07-13T02:11:54.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:54 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:11:54.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:54 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-13T02:11:54.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:54 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-13T02:11:54.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:54 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-13T02:11:54.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:54 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-13T02:11:54.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:54 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-13T02:11:54.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:54 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-13T02:11:54.272 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:54 smithi032 ceph-mon[116810]: from='client.? 172.21.15.191:0/60220453' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0386ce92-f17d-4b67-ad4e-df60b9ee4438"}]: dispatch 2023-07-13T02:11:54.272 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:54 smithi032 ceph-mon[116810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0386ce92-f17d-4b67-ad4e-df60b9ee4438"}]: dispatch 2023-07-13T02:11:54.272 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:54 smithi032 ceph-mon[116810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0386ce92-f17d-4b67-ad4e-df60b9ee4438"}]': finished 2023-07-13T02:11:54.272 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:54 smithi032 ceph-mon[116810]: osdmap e11: 7 total, 0 up, 7 in 2023-07-13T02:11:54.272 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:54 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:11:54.273 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:54 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-13T02:11:54.273 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:54 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-13T02:11:54.273 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:54 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-13T02:11:54.273 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:54 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-13T02:11:54.273 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:54 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-13T02:11:54.273 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:54 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-13T02:11:54.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:54 smithi191 ceph-mon[108588]: from='client.? 172.21.15.191:0/60220453' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0386ce92-f17d-4b67-ad4e-df60b9ee4438"}]: dispatch 2023-07-13T02:11:54.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:54 smithi191 ceph-mon[108588]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0386ce92-f17d-4b67-ad4e-df60b9ee4438"}]: dispatch 2023-07-13T02:11:54.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:54 smithi191 ceph-mon[108588]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0386ce92-f17d-4b67-ad4e-df60b9ee4438"}]': finished 2023-07-13T02:11:54.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:54 smithi191 ceph-mon[108588]: osdmap e11: 7 total, 0 up, 7 in 2023-07-13T02:11:54.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:54 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:11:54.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:54 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-13T02:11:54.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:54 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-13T02:11:54.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:54 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-13T02:11:54.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:54 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-13T02:11:54.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:54 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-13T02:11:54.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:54 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-13T02:11:54.485 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:54 smithi191 bash[122994]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:54.485 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:54 smithi191 bash[122994]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:54.485 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:54 smithi191 bash[122994]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:11:54.485 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:54 smithi191 bash[122994]: --> Failed to activate any OSD(s) 2023-07-13T02:11:54.639 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:11:54 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Failed with result 'exit-code'. 2023-07-13T02:11:54.978 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:54 smithi191 bash[123110]: de119892edc9195f7a5fa612f9ff659c8c0638bb19df55013e375cbd4375adc8 2023-07-13T02:11:54.978 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:54 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-4[123130]: 2023-07-13T02:11:54.722+0000 7fd2d1122640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:11:54.979 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:54 smithi191 systemd[1]: Started Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:54.979 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:54 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:11:55.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:55 smithi191 ceph-mon[108588]: pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:55.232 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:55 smithi191 ceph-mon[108588]: from='client.? 172.21.15.191:0/4132007274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:11:55.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:55 smithi032 ceph-mon[104859]: pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:55.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:55 smithi032 ceph-mon[104859]: from='client.? 172.21.15.191:0/4132007274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:11:55.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:55 smithi032 ceph-mon[116810]: pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:55.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:55 smithi032 ceph-mon[116810]: from='client.? 172.21.15.191:0/4132007274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:11:56.170 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:55 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:11:56.171 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:55 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Scheduled restart job, restart counter is at 5. 2023-07-13T02:11:56.171 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:55 smithi032 systemd[1]: Stopped Ceph osd.2 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:56.171 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:55 smithi032 systemd[1]: Starting Ceph osd.2 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:11:56.483 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:11:56 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Failed with result 'exit-code'. 2023-07-13T02:11:56.889 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:56 smithi032 bash[148270]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:56.889 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:56 smithi032 bash[148270]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:56.889 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:56 smithi032 bash[148270]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:11:56.889 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:56 smithi032 bash[148270]: --> Failed to activate any OSD(s) 2023-07-13T02:11:57.389 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:57 smithi032 bash[148356]: c3acc9ca095ecaeed46fb7d9e1a7c9e340d0e5e834e7ed952eaa71cd8c91cd4c 2023-07-13T02:11:57.390 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:57 smithi032 ceph-osd[148375]: -- 172.21.15.32:0/3588887685 <== mon.2 v2:172.21.15.32:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560de0aeed00 con 0x560de0abf400 2023-07-13T02:11:57.390 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:57 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-2[148371]: 2023-07-13T02:11:57.144+0000 7f90d474a640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:11:57.390 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:57 smithi032 systemd[1]: Started Ceph osd.2 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:57.808 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:57 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:11:58.118 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:57 smithi032 ceph-mon[116810]: pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:58.119 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:57 smithi032 ceph-mon[104859]: pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:58.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:57 smithi191 ceph-mon[108588]: pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:11:58.233 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:11:57 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:11:58.234 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:11:57 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Scheduled restart job, restart counter is at 1. 2023-07-13T02:11:58.234 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:11:57 smithi191 systemd[1]: Stopped Ceph osd.5 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:58.234 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:11:57 smithi191 systemd[1]: Starting Ceph osd.5 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:11:58.889 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:11:58 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Failed with result 'exit-code'. 2023-07-13T02:11:58.983 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:11:58 smithi191 bash[124435]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:58.983 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:11:58 smithi191 bash[124435]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:58.983 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:11:58 smithi191 bash[124435]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:11:58.983 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:11:58 smithi191 bash[124435]: --> Failed to activate any OSD(s) 2023-07-13T02:11:59.145 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:58 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:11:59.145 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:58 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Scheduled restart job, restart counter is at 8. 2023-07-13T02:11:59.145 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:58 smithi032 systemd[1]: Stopped Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:59.145 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:58 smithi032 systemd[1]: Starting Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:11:59.732 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:11:59 smithi191 bash[124572]: ab199250e3f00dcf2ef662c148f290da11da9ada6aa67f5040b6c792676bb67b 2023-07-13T02:11:59.733 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:11:59 smithi191 systemd[1]: Started Ceph osd.5 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:11:59.733 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:11:59 smithi191 ceph-osd[124670]: -- 172.21.15.191:0/3394570314 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c8b0b7cd00 con 0x55c8b0b4d400 2023-07-13T02:11:59.733 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:11:59 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-5[124665]: 2023-07-13T02:11:59.537+0000 7f1dc5683640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:11:59.892 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:59 smithi032 bash[148571]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:59.893 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:59 smithi032 bash[148571]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:11:59.893 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:59 smithi032 bash[148571]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:11:59.893 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:11:59 smithi032 bash[148571]: --> Failed to activate any OSD(s) 2023-07-13T02:12:00.036 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:11:59 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:12:00.036 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:59 smithi191 ceph-mon[108588]: pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:00.036 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:59 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-13T02:12:00.037 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:11:59 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:12:00.310 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:59 smithi032 ceph-mon[104859]: pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:00.311 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:59 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-13T02:12:00.311 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:11:59 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:12:00.311 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:59 smithi032 ceph-mon[116810]: pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:00.311 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:59 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-13T02:12:00.311 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:11:59 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:12:00.312 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:00 smithi032 ceph-osd[148674]: -- 172.21.15.32:0/1997453299 <== mon.2 v2:172.21.15.32:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556df2434d00 con 0x556df2405000 2023-07-13T02:12:00.312 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:00 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-0[148670]: 2023-07-13T02:12:00.072+0000 7f632d41c640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:12:00.312 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:00 smithi032 bash[148656]: a0e9fdf06153659a8d6345e78d266c8d690ea67fe54aed8ca1fd7bf65626841f 2023-07-13T02:12:00.312 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:00 smithi032 systemd[1]: Started Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:00.312 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:00 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:12:01.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:01 smithi032 ceph-mon[104859]: Deploying daemon osd.6 on smithi191 2023-07-13T02:12:01.361 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:01 smithi032 ceph-mon[116810]: Deploying daemon osd.6 on smithi191 2023-07-13T02:12:01.447 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:01 smithi191 ceph-mon[108588]: Deploying daemon osd.6 on smithi191 2023-07-13T02:12:01.639 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:01 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Failed with result 'exit-code'. 2023-07-13T02:12:01.732 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:01 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Failed with result 'exit-code'. 2023-07-13T02:12:02.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:02 smithi032 ceph-mon[104859]: pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:02.389 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:02 smithi032 ceph-mon[116810]: pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:02.412 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:02 smithi191 ceph-mon[108588]: pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:02.664 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:12:02 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.3.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:12:02.665 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:12:02 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.3.service: Scheduled restart job, restart counter is at 4. 2023-07-13T02:12:02.665 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:12:02 smithi032 systemd[1]: Stopped Ceph osd.3 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:02.665 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:12:02 smithi032 systemd[1]: Starting Ceph osd.3 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:12:03.389 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:12:02 smithi032 bash[148865]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:03.389 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:12:02 smithi032 bash[148865]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:03.389 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:12:02 smithi032 bash[148865]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:12:03.389 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:12:02 smithi032 bash[148865]: --> Failed to activate any OSD(s) 2023-07-13T02:12:03.889 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:12:03 smithi032 ceph-osd[148969]: -- 172.21.15.32:0/4024441287 <== mon.2 v2:172.21.15.32:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f1307dad00 con 0x55f1307ab000 2023-07-13T02:12:03.890 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:12:03 smithi032 bash[148951]: b0192cbd7e6d1f156d2176eeb1c4417fdba38657898d64ca1223320b72e5534f 2023-07-13T02:12:03.890 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:12:03 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-3[148965]: 2023-07-13T02:12:03.635+0000 7f8333e15640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:12:03.890 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:12:03 smithi032 systemd[1]: Started Ceph osd.3 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:04.293 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:12:03 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.3.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:12:04.294 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:04 smithi032 ceph-mon[116810]: pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:04.294 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:04 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:04.294 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:04 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:12:04.294 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:04 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:04.295 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:04 smithi032 ceph-mon[104859]: pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:04.295 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:04 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:04.295 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:04 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:12:04.295 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:04 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:04.473 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:04 smithi191 ceph-mon[108588]: pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:04.474 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:04 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:04.474 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:04 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:12:04.474 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:04 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:04.578 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:12:04 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:12:04.578 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:12:04 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Scheduled restart job, restart counter is at 7. 2023-07-13T02:12:04.578 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:12:04 smithi032 systemd[1]: Stopped Ceph osd.1 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:04.579 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:12:04 smithi032 systemd[1]: Starting Ceph osd.1 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:12:05.139 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:12:05 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.3.service: Failed with result 'exit-code'. 2023-07-13T02:12:05.139 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:12:04 smithi032 bash[149144]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:05.140 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:12:04 smithi032 bash[149144]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:05.140 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:12:04 smithi032 bash[149144]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:12:05.140 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:12:04 smithi032 bash[149144]: --> Failed to activate any OSD(s) 2023-07-13T02:12:05.518 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 6 on host 'smithi191' 2023-07-13T02:12:05.889 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:12:05 smithi032 ceph-osd[149273]: -- 172.21.15.32:0/1669347274 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bb1ac28d00 con 0x55bb1abf8c00 2023-07-13T02:12:05.890 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:12:05 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-1[149269]: 2023-07-13T02:12:05.666+0000 7f73b885d640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:12:05.890 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:12:05 smithi032 bash[149254]: ea7b8ed7b318379ff570c1e8dcff0a6a1a1de112f361ce9b09a8a923624d7f69 2023-07-13T02:12:05.890 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:12:05 smithi032 systemd[1]: Started Ceph osd.1 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:05.911 DEBUG:teuthology.orchestra.run.smithi191:osd.6> sudo journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service 2023-07-13T02:12:05.914 INFO:tasks.cephadm:Deploying osd.7 on smithi191 with /dev/nvme1n1... 2023-07-13T02:12:05.914 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph-volume lvm zap /dev/nvme1n1 2023-07-13T02:12:05.944 INFO:journalctl@ceph.osd.6.smithi191.stdout:-- Logs begin at Thu 2023-07-13 01:53:22 UTC. -- 2023-07-13T02:12:06.315 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:12:05 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:12:06.316 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:06 smithi032 ceph-mon[116810]: pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:06.316 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:06 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:06.316 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:06 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:06.316 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:06 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:06.316 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:06 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:06.317 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:06 smithi032 ceph-mon[104859]: pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:06.317 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:06 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:06.317 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:06 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:06.317 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:06 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:06.317 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:06 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:06.388 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.b/config 2023-07-13T02:12:06.472 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:06 smithi191 ceph-mon[108588]: pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:06.472 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:06 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:06.473 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:06 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:06.473 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:06 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:06.473 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:06 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:06.473 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:06 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:12:06.473 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:06 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Scheduled restart job, restart counter is at 3. 2023-07-13T02:12:06.473 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:06 smithi191 systemd[1]: Stopped Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:06.474 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:06 smithi191 systemd[1]: Starting Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:12:07.389 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:12:07 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Failed with result 'exit-code'. 2023-07-13T02:12:07.471 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:07 smithi191 bash[126163]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:07.472 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:07 smithi191 bash[126163]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:07.472 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:07 smithi191 bash[126163]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:12:07.472 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:07 smithi191 bash[126163]: --> Failed to activate any OSD(s) 2023-07-13T02:12:07.975 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/nvme1n1 2023-07-13T02:12:07.975 INFO:teuthology.orchestra.run.smithi191.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2023-07-13T02:12:07.976 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme1n1 bs=1M count=10 conv=fsync 2023-07-13T02:12:07.976 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2023-07-13T02:12:07.976 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2023-07-13T02:12:07.976 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0138253 s, 758 MB/s 2023-07-13T02:12:07.977 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2023-07-13T02:12:08.149 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:08 smithi191 ceph-mon[108588]: pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:08.149 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:08 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:08.358 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph orch daemon add osd smithi191:/dev/nvme1n1 2023-07-13T02:12:08.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:08 smithi032 ceph-mon[104859]: pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:08.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:08 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:08.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:08 smithi032 ceph-mon[116810]: pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:08.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:08 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:08.471 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:08 smithi191 bash[128455]: bafff69679b77a1dcbf3d2c580b8f5c46a5e0a6d4049c99031f125f780382538 2023-07-13T02:12:08.472 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:08 smithi191 ceph-osd[128495]: -- 172.21.15.191:0/2590865872 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56474e652d00 con 0x56474e622c00 2023-07-13T02:12:08.472 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:08 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-4[128479]: 2023-07-13T02:12:08.145+0000 7fb3d6064640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:12:08.472 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:08 smithi191 systemd[1]: Started Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:08.648 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.b/config 2023-07-13T02:12:08.732 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:08 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:12:08.894 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:12:08 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:12:08.894 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:12:08 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Scheduled restart job, restart counter is at 6. 2023-07-13T02:12:08.894 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:12:08 smithi032 systemd[1]: Stopped Ceph osd.2 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:08.894 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:12:08 smithi032 systemd[1]: Starting Ceph osd.2 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:12:09.182 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[116810]: Detected new or changed devices on smithi191 2023-07-13T02:12:09.183 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:09.183 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:12:09.183 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:12:09.183 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:12:09.183 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[116810]: Adjusting osd_memory_target on smithi191 to 5707M 2023-07-13T02:12:09.183 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:09.184 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:12:09.184 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:12:09.184 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[116810]: pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:09.184 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[116810]: pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:09.184 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:09.184 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:12:09.184 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:12:09.185 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:12:09.185 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:09.185 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[104859]: Detected new or changed devices on smithi191 2023-07-13T02:12:09.185 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:09.185 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:12:09.185 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:12:09.185 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:12:09.186 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[104859]: Adjusting osd_memory_target on smithi191 to 5707M 2023-07-13T02:12:09.186 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:09.186 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:12:09.186 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:12:09.186 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[104859]: pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:09.186 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[104859]: pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:09.186 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:09.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:12:09.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:12:09.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:12:09.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:09.386 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:09 smithi191 ceph-mon[108588]: Detected new or changed devices on smithi191 2023-07-13T02:12:09.386 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:09 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:09.386 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:09 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:12:09.386 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:09 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:12:09.387 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:09 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:12:09.387 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:09 smithi191 ceph-mon[108588]: Adjusting osd_memory_target on smithi191 to 5707M 2023-07-13T02:12:09.387 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:09 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:09.387 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:09 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:12:09.387 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:09 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:12:09.387 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:09 smithi191 ceph-mon[108588]: pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:09.387 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:09 smithi191 ceph-mon[108588]: pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:09.388 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:09 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:09.388 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:09 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:12:09.388 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:09 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:12:09.388 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:09 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:12:09.388 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:09 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:09.639 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:12:09 smithi032 bash[149541]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:09.640 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:12:09 smithi032 bash[149541]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:09.640 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:12:09 smithi032 bash[149541]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:12:09.640 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:12:09 smithi032 bash[149541]: --> Failed to activate any OSD(s) 2023-07-13T02:12:10.097 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-osd[149647]: -- 172.21.15.32:0/2874779168 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55646324ad00 con 0x55646321ac00 2023-07-13T02:12:10.097 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:12:09 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-2[149643]: 2023-07-13T02:12:09.866+0000 7f638410c640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:12:10.098 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:12:09 smithi032 bash[149629]: d9ec932f67bbfb6d796eb13305e91a6ede13a0793adc62353606d8d1a1f2c3f9 2023-07-13T02:12:10.098 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:12:09 smithi032 systemd[1]: Started Ceph osd.2 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:10.098 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:09 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Failed with result 'exit-code'. 2023-07-13T02:12:10.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:10 smithi032 ceph-mon[104859]: pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:10.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:10 smithi032 ceph-mon[104859]: pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:10.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:10 smithi032 ceph-mon[104859]: Health check update: 7 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:12:10.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:10 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:12:10.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:10 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:12:10.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:10 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:12:10.390 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:12:10 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:12:10.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:10 smithi032 ceph-mon[116810]: pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:10.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:10 smithi032 ceph-mon[116810]: pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:10.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:10 smithi032 ceph-mon[116810]: Health check update: 7 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:12:10.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:10 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:12:10.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:10 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:12:10.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:10 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:12:10.472 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:10 smithi191 ceph-mon[108588]: pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:10.472 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:10 smithi191 ceph-mon[108588]: pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:10.472 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:10 smithi191 ceph-mon[108588]: Health check update: 7 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:12:10.472 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:10 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-13T02:12:10.473 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:10 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-13T02:12:10.473 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:10 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:12:11.276 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:11 smithi032 ceph-mon[104859]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:12:11.276 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:11 smithi032 ceph-mon[116810]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:12:11.472 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:11 smithi191 ceph-mon[108588]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-13T02:12:11.628 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:11 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:12:11.629 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:11 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Scheduled restart job, restart counter is at 9. 2023-07-13T02:12:11.629 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:11 smithi032 systemd[1]: Stopped Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:11.629 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:11 smithi032 systemd[1]: Starting Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:12:11.629 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:12:11 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Failed with result 'exit-code'. 2023-07-13T02:12:11.723 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:11 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:12:11.723 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:11 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Scheduled restart job, restart counter is at 2. 2023-07-13T02:12:11.723 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:11 smithi191 systemd[1]: Stopped Ceph osd.5 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:11.723 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:11 smithi191 systemd[1]: Starting Ceph osd.5 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:12:12.389 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:11 smithi032 bash[149844]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:12.389 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:11 smithi032 bash[149844]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:12.390 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:11 smithi032 bash[149844]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:12:12.390 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:11 smithi032 bash[149844]: --> Failed to activate any OSD(s) 2023-07-13T02:12:12.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[116810]: pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:12.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[116810]: from='client.? 172.21.15.191:0/2056663306' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7aecad62-25d4-4977-a48d-58db70dc2976"}]: dispatch 2023-07-13T02:12:12.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[116810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7aecad62-25d4-4977-a48d-58db70dc2976"}]: dispatch 2023-07-13T02:12:12.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[116810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7aecad62-25d4-4977-a48d-58db70dc2976"}]': finished 2023-07-13T02:12:12.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[116810]: osdmap e12: 8 total, 0 up, 8 in 2023-07-13T02:12:12.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:12:12.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-13T02:12:12.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-13T02:12:12.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-13T02:12:12.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-13T02:12:12.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-13T02:12:12.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-13T02:12:12.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-13T02:12:12.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[104859]: pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:12.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[104859]: from='client.? 172.21.15.191:0/2056663306' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7aecad62-25d4-4977-a48d-58db70dc2976"}]: dispatch 2023-07-13T02:12:12.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[104859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7aecad62-25d4-4977-a48d-58db70dc2976"}]: dispatch 2023-07-13T02:12:12.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[104859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7aecad62-25d4-4977-a48d-58db70dc2976"}]': finished 2023-07-13T02:12:12.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[104859]: osdmap e12: 8 total, 0 up, 8 in 2023-07-13T02:12:12.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:12:12.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-13T02:12:12.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-13T02:12:12.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-13T02:12:12.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-13T02:12:12.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-13T02:12:12.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-13T02:12:12.395 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-13T02:12:12.472 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:12 smithi191 bash[129146]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:12.472 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:12 smithi191 bash[129146]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:12.472 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:12 smithi191 bash[129146]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:12:12.472 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:12 smithi191 bash[129146]: --> Failed to activate any OSD(s) 2023-07-13T02:12:12.473 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:12 smithi191 ceph-mon[108588]: pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:12.473 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:12 smithi191 ceph-mon[108588]: from='client.? 172.21.15.191:0/2056663306' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7aecad62-25d4-4977-a48d-58db70dc2976"}]: dispatch 2023-07-13T02:12:12.473 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:12 smithi191 ceph-mon[108588]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7aecad62-25d4-4977-a48d-58db70dc2976"}]: dispatch 2023-07-13T02:12:12.474 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:12 smithi191 ceph-mon[108588]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7aecad62-25d4-4977-a48d-58db70dc2976"}]': finished 2023-07-13T02:12:12.474 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:12 smithi191 ceph-mon[108588]: osdmap e12: 8 total, 0 up, 8 in 2023-07-13T02:12:12.474 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:12 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-13T02:12:12.474 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:12 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-13T02:12:12.475 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:12 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-13T02:12:12.475 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:12 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-13T02:12:12.475 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:12 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-13T02:12:12.476 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:12 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-13T02:12:12.476 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:12 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-13T02:12:12.476 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:12 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-13T02:12:12.917 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-osd[149949]: -- 172.21.15.32:0/1134732887 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5643d9b3ed00 con 0x5643d9b0f400 2023-07-13T02:12:12.917 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:12 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-0[149945]: 2023-07-13T02:12:12.635+0000 7f7e57213640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:12:12.917 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:12 smithi032 bash[149930]: b6a1bc588cce04a42aad13ada5b7b167b9a3e3e9d3f939d164dab2552f3a5fde 2023-07-13T02:12:12.917 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:12 smithi032 systemd[1]: Started Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:12.918 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:12 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:12:13.109 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:12 smithi191 bash[129262]: 08bdd116e76bb967d25f74535f8bc662de3ee66c0daa3e746b479928185f1ad7 2023-07-13T02:12:13.110 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:12 smithi191 systemd[1]: Started Ceph osd.5 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:13.110 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:12 smithi191 ceph-osd[129301]: -- 172.21.15.191:0/230115851 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f310bf0d00 con 0x55f310bc1000 2023-07-13T02:12:13.110 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:12 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-5[129297]: 2023-07-13T02:12:12.800+0000 7febdb1c3640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:12:13.110 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:13 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:12:13.287 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:13 smithi032 ceph-mon[116810]: from='client.? 172.21.15.191:0/3989978412' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:12:13.288 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:13 smithi032 ceph-mon[104859]: from='client.? 172.21.15.191:0/3989978412' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:12:13.390 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:13 smithi191 ceph-mon[108588]: from='client.? 172.21.15.191:0/3989978412' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-13T02:12:14.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:14 smithi032 ceph-mon[104859]: pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:14.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:14 smithi032 ceph-mon[116810]: pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:14.391 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:13 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Failed with result 'exit-code'. 2023-07-13T02:12:14.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:14 smithi191 ceph-mon[108588]: pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:14.483 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:14 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Failed with result 'exit-code'. 2023-07-13T02:12:15.639 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:12:15 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.3.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:12:15.640 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:12:15 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.3.service: Scheduled restart job, restart counter is at 5. 2023-07-13T02:12:15.640 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:12:15 smithi032 systemd[1]: Stopped Ceph osd.3 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:15.640 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:12:15 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.3.service: Start request repeated too quickly. 2023-07-13T02:12:15.640 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:12:15 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.3.service: Failed with result 'exit-code'. 2023-07-13T02:12:15.640 INFO:journalctl@ceph.osd.3.smithi032.stdout:Jul 13 02:12:15 smithi032 systemd[1]: Failed to start Ceph osd.3 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:15.737 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:15 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:12:15.738 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:15 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Scheduled restart job, restart counter is at 1. 2023-07-13T02:12:15.738 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:15 smithi191 systemd[1]: Stopped Ceph osd.6 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:15.738 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:15 smithi191 systemd[1]: Starting Ceph osd.6 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:12:16.336 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:16 smithi191 ceph-mon[108588]: pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:16.337 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:16 smithi191 bash[130481]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:16.337 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:16 smithi191 bash[130481]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:16.337 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:16 smithi191 bash[130481]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:12:16.338 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:16 smithi191 bash[130481]: --> Failed to activate any OSD(s) 2023-07-13T02:12:16.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:16 smithi032 ceph-mon[104859]: pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:16.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:16 smithi032 ceph-mon[116810]: pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:16.889 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:16 smithi191 ceph-osd[130700]: -- 172.21.15.191:0/2555631592 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5593b2b66d00 con 0x5593b2b37400 2023-07-13T02:12:16.890 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:16 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-6[130696]: 2023-07-13T02:12:16.862+0000 7efe91183640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:12:17.201 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:16 smithi191 bash[130662]: 93bee899f16ca5af6a83039d5ce94546d9150e8124a328904b148d346c33b678 2023-07-13T02:12:17.201 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:16 smithi191 systemd[1]: Started Ceph osd.6 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:17.202 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:17 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:12:17.639 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:12:17 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:12:17.640 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:12:17 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Scheduled restart job, restart counter is at 8. 2023-07-13T02:12:17.640 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:12:17 smithi032 systemd[1]: Stopped Ceph osd.1 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:17.640 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:12:17 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Start request repeated too quickly. 2023-07-13T02:12:17.640 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:12:17 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service: Failed with result 'exit-code'. 2023-07-13T02:12:17.640 INFO:journalctl@ceph.osd.1.smithi032.stdout:Jul 13 02:12:17 smithi032 systemd[1]: Failed to start Ceph osd.1 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:18.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:18 smithi191 ceph-mon[108588]: pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:18.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:18 smithi032 ceph-mon[104859]: pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:18.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:18 smithi032 ceph-mon[116810]: pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:18.589 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:18 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Failed with result 'exit-code'. 2023-07-13T02:12:19.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:19 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-13T02:12:19.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:19 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:12:19.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:19 smithi032 ceph-mon[104859]: Deploying daemon osd.7 on smithi191 2023-07-13T02:12:19.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:19 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-13T02:12:19.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:19 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:12:19.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:19 smithi032 ceph-mon[116810]: Deploying daemon osd.7 on smithi191 2023-07-13T02:12:19.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:19 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-13T02:12:19.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:19 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:12:19.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:19 smithi191 ceph-mon[108588]: Deploying daemon osd.7 on smithi191 2023-07-13T02:12:20.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:20 smithi191 ceph-mon[108588]: pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:20.234 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:19 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:12:20.234 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:19 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Scheduled restart job, restart counter is at 4. 2023-07-13T02:12:20.234 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:19 smithi191 systemd[1]: Stopped Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:20.234 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:19 smithi191 systemd[1]: Starting Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:12:20.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:20 smithi032 ceph-mon[104859]: pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:20.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:20 smithi032 ceph-mon[116810]: pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:20.840 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:20 smithi191 bash[131210]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:20.840 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:20 smithi191 bash[131210]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:20.840 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:20 smithi191 bash[131210]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:12:20.840 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:20 smithi191 bash[131210]: --> Failed to activate any OSD(s) 2023-07-13T02:12:21.733 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:21 smithi191 ceph-osd[131463]: -- 172.21.15.191:0/2585174751 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5590d7f36d00 con 0x5590d7f07000 2023-07-13T02:12:21.733 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:21 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-4[131456]: 2023-07-13T02:12:21.548+0000 7fe697605640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:12:21.733 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:21 smithi191 bash[131426]: 8838cb6fe544f440de1edb98ce8a89649ed8e3f3080c836c4b0868e6afd3558b 2023-07-13T02:12:21.733 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:21 smithi191 systemd[1]: Started Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:21.889 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:12:21 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:12:21.890 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:12:21 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Scheduled restart job, restart counter is at 7. 2023-07-13T02:12:21.890 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:12:21 smithi032 systemd[1]: Stopped Ceph osd.2 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:21.890 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:12:21 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Start request repeated too quickly. 2023-07-13T02:12:21.890 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:12:21 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service: Failed with result 'exit-code'. 2023-07-13T02:12:21.890 INFO:journalctl@ceph.osd.2.smithi032.stdout:Jul 13 02:12:21 smithi032 systemd[1]: Failed to start Ceph osd.2 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:22.124 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:21 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:12:22.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:22 smithi032 ceph-mon[104859]: pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:22.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:22 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:22.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:22 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:12:22.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:22 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:22.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:22 smithi032 ceph-mon[116810]: pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:22.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:22 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:22.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:22 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:12:22.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:22 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:22.433 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:22 smithi191 ceph-mon[108588]: pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:22.433 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:22 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:22.433 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:22 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:12:22.433 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:22 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:23.483 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:23 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Failed with result 'exit-code'. 2023-07-13T02:12:24.110 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:24 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:12:24.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:24 smithi032 ceph-mon[104859]: pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:24.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:24 smithi032 ceph-mon[116810]: pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:24.390 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:24 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:12:24.390 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:24 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Scheduled restart job, restart counter is at 10. 2023-07-13T02:12:24.391 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:24 smithi032 systemd[1]: Stopped Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:24.391 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:24 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Start request repeated too quickly. 2023-07-13T02:12:24.391 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:24 smithi032 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service: Failed with result 'exit-code'. 2023-07-13T02:12:24.391 INFO:journalctl@ceph.osd.0.smithi032.stdout:Jul 13 02:12:24 smithi032 systemd[1]: Failed to start Ceph osd.0 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:24.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:24 smithi191 ceph-mon[108588]: pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:24.452 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:24 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Scheduled restart job, restart counter is at 3. 2023-07-13T02:12:24.452 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:24 smithi191 systemd[1]: Stopped Ceph osd.5 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:24.452 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:24 smithi191 systemd[1]: Starting Ceph osd.5 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:12:25.083 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 7 on host 'smithi191' 2023-07-13T02:12:25.112 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:24 smithi191 bash[132305]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:25.112 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:24 smithi191 bash[132305]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:25.113 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:24 smithi191 bash[132305]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:12:25.113 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:24 smithi191 bash[132305]: --> Failed to activate any OSD(s) 2023-07-13T02:12:25.565 DEBUG:teuthology.orchestra.run.smithi191:osd.7> sudo journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.7.service 2023-07-13T02:12:25.567 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2023-07-13T02:12:25.568 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:12:25.602 INFO:journalctl@ceph.osd.7.smithi191.stdout:-- Logs begin at Thu 2023-07-13 01:53:22 UTC. -- 2023-07-13T02:12:25.817 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:12:25.890 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[104859]: pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:25.890 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:25.890 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:25.891 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:12:25.891 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:12:25.891 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[104859]: pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:25.892 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[104859]: pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:25.892 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:25.892 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:25.893 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:25.893 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:12:25.893 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[104859]: Health check update: 8 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:12:25.894 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[116810]: pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:25.894 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:25.894 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:25.895 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:12:25.895 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:12:25.895 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[116810]: pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:25.895 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[116810]: pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:25.895 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:25.896 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:25.896 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:25.896 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:12:25.896 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:25 smithi032 ceph-mon[116810]: Health check update: 8 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:12:25.983 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:25 smithi191 ceph-osd[132657]: -- 172.21.15.191:0/3813794853 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557533066d00 con 0x557533037400 2023-07-13T02:12:25.983 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:25 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-5[132646]: 2023-07-13T02:12:25.761+0000 7fd612bed640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:12:25.983 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:25 smithi191 bash[132578]: 2e2e859ebe794960c7b5c02cf446c9af9e163b2190f0144ab3f8ad8991504e01 2023-07-13T02:12:25.984 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:25 smithi191 systemd[1]: Started Ceph osd.5 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:25.984 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:25 smithi191 ceph-mon[108588]: pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:25.984 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:25 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:25.984 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:25 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:25.984 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:25 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:12:25.984 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:25 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:12:25.985 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:25 smithi191 ceph-mon[108588]: pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:25.985 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:25 smithi191 ceph-mon[108588]: pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:25.985 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:25 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:25.985 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:25 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:25.985 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:25 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:25.985 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:25 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:12:25.985 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:25 smithi191 ceph-mon[108588]: Health check update: 8 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-07-13T02:12:26.336 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:26 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:12:26.929 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:12:27.294 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:12:27.768 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:27 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Failed with result 'exit-code'. 2023-07-13T02:12:28.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:27 smithi032 ceph-mon[104859]: pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:28.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:27 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2080586465' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:28.140 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:27 smithi032 ceph-mon[116810]: pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:28.140 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:27 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2080586465' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:28.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:27 smithi191 ceph-mon[108588]: pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:28.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:27 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/2080586465' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:28.295 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:12:28.541 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:12:28.891 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:28 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:12:28.891 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:28 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Scheduled restart job, restart counter is at 2. 2023-07-13T02:12:28.892 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:28 smithi191 systemd[1]: Stopped Ceph osd.6 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:28.892 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:28 smithi191 systemd[1]: Starting Ceph osd.6 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:12:29.058 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[116810]: Detected new or changed devices on smithi191 2023-07-13T02:12:29.058 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:29.059 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:29.059 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:12:29.059 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:12:29.059 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:12:29.059 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:12:29.059 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[116810]: Adjusting osd_memory_target on smithi191 to 4280M 2023-07-13T02:12:29.059 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:29.059 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:12:29.060 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:12:29.060 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[116810]: pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:29.060 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[116810]: pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:29.060 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:29.060 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[104859]: Detected new or changed devices on smithi191 2023-07-13T02:12:29.061 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:29.061 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:29.061 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:12:29.061 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:12:29.061 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:12:29.061 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:12:29.061 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[104859]: Adjusting osd_memory_target on smithi191 to 4280M 2023-07-13T02:12:29.062 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:29.062 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:12:29.062 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:12:29.062 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[104859]: pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:29.062 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[104859]: pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:29.063 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:28 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:29.233 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:29 smithi191 bash[135154]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:29.233 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:29 smithi191 bash[135154]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:29.233 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:29 smithi191 bash[135154]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:12:29.233 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:29 smithi191 bash[135154]: --> Failed to activate any OSD(s) 2023-07-13T02:12:29.234 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:28 smithi191 ceph-mon[108588]: Detected new or changed devices on smithi191 2023-07-13T02:12:29.234 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:28 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:29.234 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:28 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:29.234 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:28 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:12:29.234 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:28 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:12:29.234 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:28 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:12:29.235 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:28 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-13T02:12:29.235 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:28 smithi191 ceph-mon[108588]: Adjusting osd_memory_target on smithi191 to 4280M 2023-07-13T02:12:29.235 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:28 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:29.235 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:28 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:12:29.235 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:28 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:12:29.235 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:28 smithi191 ceph-mon[108588]: pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:29.235 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:28 smithi191 ceph-mon[108588]: pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:29.236 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:28 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:12:29.681 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:12:29.905 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:29 smithi191 bash[135240]: 6f1f7c6546bf61df775b1d5d5438dba0ca289495b3bf2aae4cd7b1992133c9db 2023-07-13T02:12:29.905 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:29 smithi191 systemd[1]: Started Ceph osd.6 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:29.906 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:29 smithi191 ceph-osd[135259]: -- 172.21.15.191:0/2772639775 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55908f35ed00 con 0x55908f32ec00 2023-07-13T02:12:29.906 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:29 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-6[135255]: 2023-07-13T02:12:29.645+0000 7f3c52f0a640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:12:29.906 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:29 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:12:30.110 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:12:30.211 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:29 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1352509083' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:30.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:29 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1352509083' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:30.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:29 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1352509083' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:31.112 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:12:31.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:31 smithi191 ceph-mon[108588]: pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:31.233 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:30 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Failed with result 'exit-code'. 2023-07-13T02:12:31.356 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:12:31.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:31 smithi032 ceph-mon[104859]: pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:31.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:31 smithi032 ceph-mon[116810]: pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:32.503 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:12:32.869 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:12:33.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:33 smithi032 ceph-mon[104859]: pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:33.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:33 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2973848521' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:33.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:33 smithi032 ceph-mon[116810]: pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:33.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:33 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2973848521' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:33.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:33 smithi191 ceph-mon[108588]: pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:33.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:33 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/2973848521' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:33.870 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:12:33.972 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:33 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:12:33.973 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:33 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Scheduled restart job, restart counter is at 5. 2023-07-13T02:12:33.973 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:33 smithi191 systemd[1]: Stopped Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:33.973 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:33 smithi191 systemd[1]: Starting Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:12:34.116 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:12:34.233 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:33 smithi191 bash[135455]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:34.233 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:33 smithi191 bash[135455]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:34.233 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:33 smithi191 bash[135455]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:12:34.233 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:33 smithi191 bash[135455]: --> Failed to activate any OSD(s) 2023-07-13T02:12:34.797 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:34 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.7.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:12:34.797 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:34 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.7.service: Scheduled restart job, restart counter is at 1. 2023-07-13T02:12:34.797 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:34 smithi191 systemd[1]: Stopped Ceph osd.7 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:34.797 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:34 smithi191 systemd[1]: Starting Ceph osd.7 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:12:34.798 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:34 smithi191 bash[135543]: 4dd5762131298e6e6b410998834a382e232a97d3f01638ec317df56b31093ad2 2023-07-13T02:12:34.798 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:34 smithi191 systemd[1]: Started Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:34.798 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:34 smithi191 ceph-osd[135608]: -- 172.21.15.191:0/2694800467 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e8ece88d00 con 0x55e8ece59000 2023-07-13T02:12:34.798 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:34 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-4[135574]: 2023-07-13T02:12:34.586+0000 7fd7fe825640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:12:34.798 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:34 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:12:35.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:35 smithi032 ceph-mon[104859]: pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:35.140 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:35 smithi032 ceph-mon[116810]: pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:35.256 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:12:35.350 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:35 smithi191 ceph-mon[108588]: pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:35.350 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:35 smithi191 bash[135654]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:35.350 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:35 smithi191 bash[135654]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:35.351 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:35 smithi191 bash[135654]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:12:35.351 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:35 smithi191 bash[135654]: --> Failed to activate any OSD(s) 2023-07-13T02:12:35.635 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:12:36.278 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:36 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1997269836' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:36.279 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:36 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Failed with result 'exit-code'. 2023-07-13T02:12:36.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:36 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1997269836' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:36.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:36 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1997269836' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:36.579 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:36 smithi191 bash[135825]: 814d9c4ade2fecff63fbf3b586343a6b0d9d87b42db10a21d06a39f709444349 2023-07-13T02:12:36.580 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:36 smithi191 systemd[1]: Started Ceph osd.7 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:36.580 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:36 smithi191 ceph-osd[135863]: -- 172.21.15.191:0/2976648258 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556f5c826d00 con 0x556f5c7f7400 2023-07-13T02:12:36.580 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:36 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-7[135857]: 2023-07-13T02:12:36.279+0000 7fdb4d463640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:12:36.580 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:36 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.7.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:12:36.636 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:12:36.880 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:12:37.332 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:37 smithi191 ceph-mon[108588]: pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:37.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:37 smithi032 ceph-mon[104859]: pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:37.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:37 smithi032 ceph-mon[116810]: pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:37.983 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:37 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.7.service: Failed with result 'exit-code'. 2023-07-13T02:12:37.983 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:37 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:12:37.983 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:37 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Scheduled restart job, restart counter is at 4. 2023-07-13T02:12:37.984 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:37 smithi191 systemd[1]: Stopped Ceph osd.5 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:37.984 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:37 smithi191 systemd[1]: Starting Ceph osd.5 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:12:37.989 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:12:38.222 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:38 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3074117086' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:38.223 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:38 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3074117086' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:38.409 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:12:38.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:38 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3074117086' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:38.483 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:38 smithi191 bash[136059]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:38.484 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:38 smithi191 bash[136059]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:38.484 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:38 smithi191 bash[136059]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:12:38.484 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:38 smithi191 bash[136059]: --> Failed to activate any OSD(s) 2023-07-13T02:12:38.983 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:38 smithi191 bash[136144]: 48031de0bb33cf93020613d294d24ceeeb345769f693ffab01def927361005b8 2023-07-13T02:12:38.983 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:38 smithi191 ceph-osd[136163]: -- 172.21.15.191:0/932783753 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d0ddeaad00 con 0x55d0dde7b400 2023-07-13T02:12:38.983 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:38 smithi191 systemd[1]: Started Ceph osd.5 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:38.984 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:38 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-5[136159]: 2023-07-13T02:12:38.822+0000 7fedbe4d9640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:12:39.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:39 smithi032 ceph-mon[104859]: pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:39.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:39 smithi032 ceph-mon[116810]: pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:39.410 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:12:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:39 smithi191 ceph-mon[108588]: pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:39.467 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:39 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:12:39.655 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:12:40.483 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:40 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Failed with result 'exit-code'. 2023-07-13T02:12:40.788 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:12:41.135 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:12:41.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:41 smithi032 ceph-mon[104859]: pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:41.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:41 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3400996370' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:41.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:41 smithi032 ceph-mon[116810]: pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:41.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:41 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3400996370' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:41.476 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:41 smithi191 ceph-mon[108588]: pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:41.477 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:41 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3400996370' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:41.477 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:40 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:12:41.477 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:40 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Scheduled restart job, restart counter is at 3. 2023-07-13T02:12:41.477 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:40 smithi191 systemd[1]: Stopped Ceph osd.6 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:41.477 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:40 smithi191 systemd[1]: Starting Ceph osd.6 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:12:41.733 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:41 smithi191 bash[136357]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:41.733 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:41 smithi191 bash[136357]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:41.733 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:41 smithi191 bash[136357]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:12:41.733 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:41 smithi191 bash[136357]: --> Failed to activate any OSD(s) 2023-07-13T02:12:42.136 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:12:42.364 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:42 smithi191 bash[136445]: 16a9b48060ae9e8eaabf3ee7ebc60fae0bc97769e16310c22e578127c3cae217 2023-07-13T02:12:42.364 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:42 smithi191 systemd[1]: Started Ceph osd.6 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:42.364 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:42 smithi191 ceph-osd[136463]: -- 172.21.15.191:0/497500155 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556c1342ad00 con 0x556c133fb000 2023-07-13T02:12:42.364 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:42 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-6[136459]: 2023-07-13T02:12:42.111+0000 7f0b4db7a640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:12:42.364 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:42 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:12:42.380 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:12:43.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:43 smithi032 ceph-mon[104859]: pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:43.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:43 smithi032 ceph-mon[116810]: pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:43.404 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:43 smithi191 ceph-mon[108588]: pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:43.519 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:12:43.733 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:43 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Failed with result 'exit-code'. 2023-07-13T02:12:43.901 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:12:44.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:44 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/676803293' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:44.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:44 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/676803293' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:44.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:44 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/676803293' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:44.902 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:12:45.149 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:12:45.270 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:45 smithi032 ceph-mon[116810]: pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:45.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:45 smithi032 ceph-mon[104859]: pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:45.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:45 smithi191 ceph-mon[108588]: pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:46.275 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:12:46.488 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:46 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:12:46.488 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:46 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Scheduled restart job, restart counter is at 6. 2023-07-13T02:12:46.488 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:46 smithi191 systemd[1]: Stopped Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:46.489 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:46 smithi191 systemd[1]: Starting Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:12:46.651 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:12:46.983 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:46 smithi191 bash[136666]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:46.983 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:46 smithi191 bash[136666]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:46.983 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:46 smithi191 bash[136666]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:12:46.984 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:46 smithi191 bash[136666]: --> Failed to activate any OSD(s) 2023-07-13T02:12:47.374 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:47 smithi191 ceph-mon[108588]: pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:47.375 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:47 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3206069248' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:47.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:47 smithi032 ceph-mon[104859]: pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:47.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:47 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3206069248' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:47.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:47 smithi032 ceph-mon[116810]: pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:47.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3206069248' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:47.652 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:12:47.733 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:47 smithi191 bash[136752]: 9839778f5e0a00ec8848d26d1a13d0df72e89a3e1dd5605dc7334828c922b3ef 2023-07-13T02:12:47.734 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:47 smithi191 systemd[1]: Started Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:47.734 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:47 smithi191 ceph-osd[136770]: -- 172.21.15.191:0/3687483164 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5649a47dcd00 con 0x5649a47ad000 2023-07-13T02:12:47.734 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:47 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-4[136766]: 2023-07-13T02:12:47.427+0000 7f3fa8ba1640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:12:47.734 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:47 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:12:47.898 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:12:47.988 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:47 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.7.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:12:47.989 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:47 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.7.service: Scheduled restart job, restart counter is at 2. 2023-07-13T02:12:47.989 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:47 smithi191 systemd[1]: Stopped Ceph osd.7 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:47.989 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:47 smithi191 systemd[1]: Starting Ceph osd.7 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:12:48.436 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:48 smithi191 bash[136888]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:48.436 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:48 smithi191 bash[136888]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:48.436 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:48 smithi191 bash[136888]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:12:48.437 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:48 smithi191 bash[136888]: --> Failed to activate any OSD(s) 2023-07-13T02:12:49.029 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:12:49.117 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:48 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Failed with result 'exit-code'. 2023-07-13T02:12:49.118 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:49 smithi191 bash[137053]: c5cb3af45cd16bce8283294e3774e90bd062e60eb749e1884259eda22fd715be 2023-07-13T02:12:49.119 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:49 smithi191 systemd[1]: Started Ceph osd.7 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:49.119 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:49 smithi191 ceph-osd[137073]: -- 172.21.15.191:0/3941626920 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bf2b58ed00 con 0x55bf2b55ec00 2023-07-13T02:12:49.119 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:49 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-7[137069]: 2023-07-13T02:12:49.044+0000 7f457cf31640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:12:49.345 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:49 smithi032 ceph-mon[116810]: pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:49.345 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:49 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3585881696' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:49.345 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:49 smithi032 ceph-mon[104859]: pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:49.345 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:49 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3585881696' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:49.400 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:12:49.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:49 smithi191 ceph-mon[108588]: pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:49.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:49 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3585881696' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:49.483 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:49 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.7.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:12:50.233 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:50 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:12:50.233 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:50 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Scheduled restart job, restart counter is at 5. 2023-07-13T02:12:50.234 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:50 smithi191 systemd[1]: Stopped Ceph osd.5 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:50.234 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:50 smithi191 systemd[1]: Starting Ceph osd.5 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:12:50.402 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:12:50.647 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:12:50.685 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:12:50 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.7.service: Failed with result 'exit-code'. 2023-07-13T02:12:50.983 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:50 smithi191 bash[137270]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:50.984 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:50 smithi191 bash[137270]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:50.984 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:50 smithi191 bash[137270]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:12:50.984 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:50 smithi191 bash[137270]: --> Failed to activate any OSD(s) 2023-07-13T02:12:51.380 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:51 smithi032 ceph-mon[116810]: pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:51.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:51 smithi032 ceph-mon[104859]: pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:51.424 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:51 smithi191 ceph-osd[137375]: -- 172.21.15.191:0/3933131656 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5606bee74d00 con 0x5606bee45400 2023-07-13T02:12:51.425 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:51 smithi191 bash[137357]: 9091c50878602af98cda8080d1a13fd20d9db6ee32e5f756baa329952b5f7e90 2023-07-13T02:12:51.425 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:51 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-5[137371]: 2023-07-13T02:12:51.352+0000 7fecb855b640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:12:51.425 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:51 smithi191 systemd[1]: Started Ceph osd.5 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:51.425 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:51 smithi191 ceph-mon[108588]: pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:51.733 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:51 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:12:51.759 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:12:52.177 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:12:52.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:52 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/2108352448' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:52.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:52 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2108352448' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:52.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:52 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2108352448' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:52.983 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:12:52 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Failed with result 'exit-code'. 2023-07-13T02:12:53.179 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:12:53.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:53 smithi032 ceph-mon[104859]: pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:53.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:53 smithi032 ceph-mon[116810]: pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:53.426 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:12:53.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:53 smithi191 ceph-mon[108588]: pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:53.747 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:53 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:12:53.747 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:53 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Scheduled restart job, restart counter is at 4. 2023-07-13T02:12:53.747 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:53 smithi191 systemd[1]: Stopped Ceph osd.6 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:53.747 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:53 smithi191 systemd[1]: Starting Ceph osd.6 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:12:54.483 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:54 smithi191 bash[137570]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:54.483 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:54 smithi191 bash[137570]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:54.484 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:54 smithi191 bash[137570]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:12:54.484 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:54 smithi191 bash[137570]: --> Failed to activate any OSD(s) 2023-07-13T02:12:54.554 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:12:54.925 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:12:54.926 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:54 smithi191 bash[137657]: 474c81d0a7513326f8de96377d49266fb1a55e2a1d778f46b990b9159c19430d 2023-07-13T02:12:54.926 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:54 smithi191 systemd[1]: Started Ceph osd.6 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:54.927 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:54 smithi191 ceph-osd[137676]: -- 172.21.15.191:0/4014268580 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fac55c8d00 con 0x55fac5599400 2023-07-13T02:12:54.927 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:54 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-6[137672]: 2023-07-13T02:12:54.659+0000 7f5a7c42f640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:12:54.927 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:54 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:12:55.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:55 smithi191 ceph-mon[108588]: pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:55.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:55 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1056698396' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:55.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:55 smithi032 ceph-mon[104859]: pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:55.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:55 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1056698396' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:55.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:55 smithi032 ceph-mon[116810]: pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:55.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:55 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1056698396' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:55.926 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:12:56.173 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:12:56.233 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:12:55 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Failed with result 'exit-code'. 2023-07-13T02:12:57.290 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:57 smithi032 ceph-mon[104859]: pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:57.291 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:57 smithi032 ceph-mon[116810]: pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:57.373 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:12:57.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:57 smithi191 ceph-mon[108588]: pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:57.750 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:12:58.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:58 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2099221454' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:58.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:58 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2099221454' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:58.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:58 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/2099221454' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:12:58.751 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:12:58.993 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:12:59.234 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:12:59 smithi191 ceph-mon[108588]: pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:59.235 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:58 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:12:59.235 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:58 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Scheduled restart job, restart counter is at 7. 2023-07-13T02:12:59.235 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:58 smithi191 systemd[1]: Stopped Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:12:59.235 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:58 smithi191 systemd[1]: Starting Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:12:59.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:12:59 smithi032 ceph-mon[104859]: pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:59.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:12:59 smithi032 ceph-mon[116810]: pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:12:59.540 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:59 smithi191 bash[137871]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:59.541 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:59 smithi191 bash[137871]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:12:59.541 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:59 smithi191 bash[137871]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:12:59.541 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:12:59 smithi191 bash[137871]: --> Failed to activate any OSD(s) 2023-07-13T02:13:00.120 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:13:00.401 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:00 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.7.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:13:00.402 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:00 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.7.service: Scheduled restart job, restart counter is at 3. 2023-07-13T02:13:00.402 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:00 smithi191 systemd[1]: Stopped Ceph osd.7 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:00.403 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:00 smithi191 systemd[1]: Starting Ceph osd.7 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:13:00.403 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:00 smithi191 bash[137956]: 1d6872f3b971d35ca0e650be0a5272b8833c4a03a380d3105b759d48bf2865f4 2023-07-13T02:13:00.404 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:00 smithi191 ceph-osd[137976]: -- 172.21.15.191:0/1545545026 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d4df4bad00 con 0x55d4df48b000 2023-07-13T02:13:00.404 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:00 smithi191 systemd[1]: Started Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:00.404 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:00 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-4[137972]: 2023-07-13T02:13:00.128+0000 7fb6581a7640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:13:00.404 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:00 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:13:00.524 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:13:01.170 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:01 smithi191 ceph-mon[108588]: pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:01.170 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:01 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/716879649' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:01.171 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:00 smithi191 bash[138073]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:13:01.171 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:00 smithi191 bash[138073]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:13:01.171 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:00 smithi191 bash[138073]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:13:01.171 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:00 smithi191 bash[138073]: --> Failed to activate any OSD(s) 2023-07-13T02:13:01.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:01 smithi032 ceph-mon[116810]: pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:01.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:01 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/716879649' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:01.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:01 smithi032 ceph-mon[104859]: pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:01.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:01 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/716879649' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:01.525 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:13:01.733 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:01 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Failed with result 'exit-code'. 2023-07-13T02:13:01.772 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:13:02.168 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:01 smithi191 bash[138253]: 138e9cfca21c8aef8df6c79136565d339180b0841adb55f6d0fb28241354f640 2023-07-13T02:13:02.169 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:01 smithi191 systemd[1]: Started Ceph osd.7 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:02.169 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:01 smithi191 ceph-osd[138272]: -- 172.21.15.191:0/2208290305 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55951be6ad00 con 0x55951be3b000 2023-07-13T02:13:02.169 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:01 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-7[138268]: 2023-07-13T02:13:01.817+0000 7f71d6a36640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:13:02.169 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:02 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.7.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:13:02.886 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:13:03.079 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:02 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:13:03.079 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:02 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Scheduled restart job, restart counter is at 6. 2023-07-13T02:13:03.079 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:02 smithi191 systemd[1]: Stopped Ceph osd.5 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:03.079 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:02 smithi191 systemd[1]: Starting Ceph osd.5 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:13:03.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:03 smithi032 ceph-mon[104859]: pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:03.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:03 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/695428718' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:03.141 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:03 smithi032 ceph-mon[116810]: pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:03.141 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:03 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/695428718' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:03.265 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:13:03.435 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:03 smithi191 ceph-mon[108588]: pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:03.435 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:03 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/695428718' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:03.436 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:03 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.7.service: Failed with result 'exit-code'. 2023-07-13T02:13:03.734 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:03 smithi191 bash[138467]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:13:03.734 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:03 smithi191 bash[138467]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:13:03.734 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:03 smithi191 bash[138467]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:13:03.734 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:03 smithi191 bash[138467]: --> Failed to activate any OSD(s) 2023-07-13T02:13:04.267 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:13:04.317 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:04 smithi191 bash[138554]: 0e67b38c46e77b1a20d7b345767ebd7b2dfbe7a8e5948700d47035f194b05474 2023-07-13T02:13:04.317 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:04 smithi191 systemd[1]: Started Ceph osd.5 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:04.318 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:04 smithi191 ceph-osd[138573]: -- 172.21.15.191:0/58141493 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d178ce4d00 con 0x55d178cb5000 2023-07-13T02:13:04.318 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:04 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-5[138569]: 2023-07-13T02:13:04.066+0000 7fcf2226d640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:13:04.318 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:04 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:13:04.511 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:13:05.375 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:05 smithi191 ceph-mon[108588]: pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:05.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:05 smithi032 ceph-mon[104859]: pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:05.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:05 smithi032 ceph-mon[116810]: pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:05.676 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:13:05.733 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:05 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Failed with result 'exit-code'. 2023-07-13T02:13:06.066 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:13:06.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:06 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3412459561' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:06.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:06 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3412459561' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:06.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:06 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3412459561' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:06.482 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:05 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:13:06.483 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:05 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Scheduled restart job, restart counter is at 5. 2023-07-13T02:13:06.483 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:05 smithi191 systemd[1]: Stopped Ceph osd.6 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:06.483 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:05 smithi191 systemd[1]: Starting Ceph osd.6 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:13:06.733 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:06 smithi191 bash[138768]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:13:06.733 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:06 smithi191 bash[138768]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:13:06.734 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:06 smithi191 bash[138768]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:13:06.734 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:06 smithi191 bash[138768]: --> Failed to activate any OSD(s) 2023-07-13T02:13:07.067 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:13:07.313 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:13:07.383 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:07 smithi191 ceph-mon[108588]: pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:07.383 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:07 smithi191 bash[138856]: 7ee781da41ee0b40c39c4e126f0d573d38d0d4617c1f516b97e8c86b6101c9ef 2023-07-13T02:13:07.383 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:07 smithi191 systemd[1]: Started Ceph osd.6 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:07.384 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:07 smithi191 ceph-osd[138875]: -- 172.21.15.191:0/1511347255 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5636af42ed00 con 0x5636af3fec00 2023-07-13T02:13:07.384 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:07 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-6[138871]: 2023-07-13T02:13:07.102+0000 7f076e105640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:13:07.384 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:07 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:13:07.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:07 smithi032 ceph-mon[104859]: pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:07.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:07 smithi032 ceph-mon[116810]: pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:08.478 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:13:08.733 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:08 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Failed with result 'exit-code'. 2023-07-13T02:13:08.915 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:13:09.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:09 smithi032 ceph-mon[104859]: pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:09.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:09 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2691357443' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:09.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:09 smithi032 ceph-mon[116810]: pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:09.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:09 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2691357443' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:09.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:09 smithi191 ceph-mon[108588]: pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:09.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:09 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/2691357443' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:09.917 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:13:10.163 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:13:11.141 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:11 smithi032 ceph-mon[104859]: pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:11.141 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:11 smithi032 ceph-mon[116810]: pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:11.266 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:13:11.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:11 smithi191 ceph-mon[108588]: pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:11.673 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:13:11.992 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:11 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:13:11.992 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:11 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Scheduled restart job, restart counter is at 8. 2023-07-13T02:13:11.992 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:11 smithi191 systemd[1]: Stopped Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:11.993 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:11 smithi191 systemd[1]: Starting Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:13:12.272 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:12 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3253990943' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:12.272 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:12 smithi191 bash[139069]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:13:12.273 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:12 smithi191 bash[139069]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:13:12.273 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:12 smithi191 bash[139069]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:13:12.273 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:12 smithi191 bash[139069]: --> Failed to activate any OSD(s) 2023-07-13T02:13:12.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:12 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3253990943' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:12.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:12 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3253990943' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:12.675 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:13:12.921 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:13:13.116 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:13 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.7.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:13:13.116 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:13 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.7.service: Scheduled restart job, restart counter is at 4. 2023-07-13T02:13:13.116 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:13 smithi191 systemd[1]: Stopped Ceph osd.7 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:13.116 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:13 smithi191 systemd[1]: Starting Ceph osd.7 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:13:13.116 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:12 smithi191 bash[139155]: 59f485e973a2678c68e72894c2343740a16e2679dc1f98c8ac76c2dbd06139d3 2023-07-13T02:13:13.117 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:12 smithi191 systemd[1]: Started Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:13.117 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:12 smithi191 ceph-osd[139174]: -- 172.21.15.191:0/3084856135 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b30d5c2d00 con 0x55b30d593400 2023-07-13T02:13:13.117 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:12 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-4[139170]: 2023-07-13T02:13:12.863+0000 7fd12f2b3640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:13:13.117 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:13 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:13:13.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:13 smithi032 ceph-mon[104859]: pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:13.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:13 smithi032 ceph-mon[116810]: pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:13.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:13 smithi191 ceph-mon[108588]: pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:13.935 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:13 smithi191 bash[139273]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:13:13.935 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:13 smithi191 bash[139273]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:13:13.935 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:13 smithi191 bash[139273]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:13:13.935 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:13 smithi191 bash[139273]: --> Failed to activate any OSD(s) 2023-07-13T02:13:13.993 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:13:14.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:14 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/119058499' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:14.288 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:14 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/119058499' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:14.289 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:14 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/119058499' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:14.323 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:13:14.632 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:14 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Failed with result 'exit-code'. 2023-07-13T02:13:14.632 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:14 smithi191 bash[139451]: ed531b9e4ca9615b6fcfd350aa407a7a2147615324deb8726e17c7751e90e016 2023-07-13T02:13:14.632 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:14 smithi191 systemd[1]: Started Ceph osd.7 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:14.633 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:14 smithi191 ceph-osd[139470]: -- 172.21.15.191:0/647410598 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5627197f8d00 con 0x5627197c8c00 2023-07-13T02:13:14.633 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:14 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-7[139466]: 2023-07-13T02:13:14.430+0000 7fbc661b2640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:13:14.633 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:14 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.7.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:13:15.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:15 smithi191 ceph-mon[108588]: pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:15.324 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:13:15.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:15 smithi032 ceph-mon[116810]: pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:15.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:15 smithi032 ceph-mon[104859]: pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:15.569 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:13:15.690 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:15 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:13:15.690 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:15 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Scheduled restart job, restart counter is at 7. 2023-07-13T02:13:15.690 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:15 smithi191 systemd[1]: Stopped Ceph osd.5 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:15.690 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:15 smithi191 systemd[1]: Starting Ceph osd.5 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:13:15.983 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:15 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.7.service: Failed with result 'exit-code'. 2023-07-13T02:13:16.483 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:16 smithi191 bash[139658]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:13:16.484 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:16 smithi191 bash[139658]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:13:16.484 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:16 smithi191 bash[139658]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:13:16.484 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:16 smithi191 bash[139658]: --> Failed to activate any OSD(s) 2023-07-13T02:13:16.689 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:13:16.933 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:16 smithi191 ceph-osd[139764]: -- 172.21.15.191:0/3973947280 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5555b0fc4d00 con 0x5555b0f95000 2023-07-13T02:13:16.934 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:16 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-5[139760]: 2023-07-13T02:13:16.674+0000 7f78ca9c3640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:13:16.934 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:16 smithi191 bash[139745]: 76be826d77d0574705335bba5b4b776d13b734da50417132a73840417532cae4 2023-07-13T02:13:16.934 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:16 smithi191 systemd[1]: Started Ceph osd.5 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:16.934 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:16 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:13:17.080 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:13:17.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:17 smithi191 ceph-mon[108588]: pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:17.234 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:17 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3700191585' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:17.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:17 smithi032 ceph-mon[104859]: pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:17.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:17 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3700191585' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:17.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:17 smithi032 ceph-mon[116810]: pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:17.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:17 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3700191585' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:18.082 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:13:18.234 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:17 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Failed with result 'exit-code'. 2023-07-13T02:13:18.328 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:13:18.756 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:18 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:13:18.756 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:18 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Scheduled restart job, restart counter is at 6. 2023-07-13T02:13:18.756 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:18 smithi191 systemd[1]: Stopped Ceph osd.6 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:18.756 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:18 smithi191 systemd[1]: Starting Ceph osd.6 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:13:19.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:19 smithi032 ceph-mon[104859]: pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:19.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:19 smithi032 ceph-mon[116810]: pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:19.429 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:13:19.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:19 smithi191 ceph-mon[108588]: pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:19.484 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:19 smithi191 bash[139959]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:13:19.484 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:19 smithi191 bash[139959]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:13:19.484 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:19 smithi191 bash[139959]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:13:19.485 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:19 smithi191 bash[139959]: --> Failed to activate any OSD(s) 2023-07-13T02:13:19.821 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:13:19.983 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:19 smithi191 bash[140044]: 2d69473e2c816ddc5dcfca2040609cab6c84af0c4c388de141f1781667e75011 2023-07-13T02:13:19.984 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:19 smithi191 systemd[1]: Started Ceph osd.6 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:19.984 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:19 smithi191 ceph-osd[140063]: -- 172.21.15.191:0/642134424 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fb8e462d00 con 0x55fb8e433400 2023-07-13T02:13:19.984 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:19 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-6[140059]: 2023-07-13T02:13:19.718+0000 7f235143f640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:13:19.984 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:19 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:13:20.130 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:20 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/19798918' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:20.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:20 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/19798918' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:20.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:20 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/19798918' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:20.822 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:13:21.067 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:13:21.233 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:21 smithi191 ceph-mon[108588]: pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:21.234 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:20 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Failed with result 'exit-code'. 2023-07-13T02:13:21.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:21 smithi032 ceph-mon[104859]: pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:21.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:21 smithi032 ceph-mon[116810]: pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:22.176 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:13:22.548 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:13:23.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:23 smithi032 ceph-mon[104859]: pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:23.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:23 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3748804946' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:23.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:23 smithi032 ceph-mon[116810]: pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:23.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:23 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3748804946' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:23.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:23 smithi191 ceph-mon[108588]: pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:23.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:23 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3748804946' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:23.549 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:13:23.794 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:13:24.959 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:13:24.984 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:24 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:13:24.984 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:24 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Scheduled restart job, restart counter is at 9. 2023-07-13T02:13:24.985 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:24 smithi191 systemd[1]: Stopped Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:24.985 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:24 smithi191 systemd[1]: Starting Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:13:25.234 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:25 smithi191 ceph-mon[108588]: pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:25.234 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:25 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3369102911' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:25.234 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:24 smithi191 bash[140259]: --> Failed to activate via raw: 'Activate' object has no attribute 'activate' 2023-07-13T02:13:25.235 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:24 smithi191 bash[140259]: --> Failed to activate via LVM: 'Activate' object has no attribute 'activate' 2023-07-13T02:13:25.235 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:24 smithi191 bash[140259]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-07-13T02:13:25.235 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:24 smithi191 bash[140259]: --> Failed to activate any OSD(s) 2023-07-13T02:13:25.270 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:25 smithi032 ceph-mon[116810]: pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:25.271 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:25 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3369102911' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:25.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:25 smithi032 ceph-mon[104859]: pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:25.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:25 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3369102911' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:25.346 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:13:25.984 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:25 smithi191 ceph-osd[140365]: -- 172.21.15.191:0/3057980578 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557b45ddad00 con 0x557b45daac00 2023-07-13T02:13:25.985 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:25 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-osd-4[140361]: 2023-07-13T02:13:25.611+0000 7fa78ab05640 -1 missing 'type' file and unable to infer osd type 2023-07-13T02:13:25.985 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:25 smithi191 bash[140346]: 3a9f81473abb9e30f3384f78cf85380f8f49a5b41fe33f7782b581326408d642 2023-07-13T02:13:25.985 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:25 smithi191 systemd[1]: Started Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:25.985 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:25 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Main process exited, code=exited, status=1/FAILURE 2023-07-13T02:13:25.986 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:25 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.7.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:13:25.986 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:25 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.7.service: Scheduled restart job, restart counter is at 5. 2023-07-13T02:13:25.987 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:25 smithi191 systemd[1]: Stopped Ceph osd.7 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:25.987 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:25 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.7.service: Start request repeated too quickly. 2023-07-13T02:13:25.987 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:25 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.7.service: Failed with result 'exit-code'. 2023-07-13T02:13:25.988 INFO:journalctl@ceph.osd.7.smithi191.stdout:Jul 13 02:13:25 smithi191 systemd[1]: Failed to start Ceph osd.7 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:26.348 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:13:26.594 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:13:27.234 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:27 smithi191 ceph-mon[108588]: pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:27.234 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:26 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Failed with result 'exit-code'. 2023-07-13T02:13:27.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:27 smithi032 ceph-mon[104859]: pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:27.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:27 smithi032 ceph-mon[116810]: pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:27.809 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:13:28.182 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:13:28.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:28 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1109154265' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:28.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:28 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:13:28.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:28 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1109154265' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:28.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:28 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:13:28.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:28 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1109154265' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:28.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:28 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:13:28.485 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:27 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:13:28.485 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:27 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Scheduled restart job, restart counter is at 8. 2023-07-13T02:13:28.485 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:27 smithi191 systemd[1]: Stopped Ceph osd.5 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:28.485 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:27 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Start request repeated too quickly. 2023-07-13T02:13:28.485 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:27 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service: Failed with result 'exit-code'. 2023-07-13T02:13:28.485 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 13 02:13:27 smithi191 systemd[1]: Failed to start Ceph osd.5 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:29.183 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:13:29.428 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:13:29.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:29 smithi191 ceph-mon[108588]: pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:29.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:29 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:13:29.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:29 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:13:29.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:29 smithi191 ceph-mon[108588]: pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:29.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:29 smithi191 ceph-mon[108588]: pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:29.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:29 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:13:29.616 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:29 smithi032 ceph-mon[116810]: pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:29.616 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:29 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:13:29.617 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:29 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:13:29.617 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:29 smithi032 ceph-mon[116810]: pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:29.617 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:29 smithi032 ceph-mon[116810]: pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:29.617 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:29 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:13:29.618 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:29 smithi032 ceph-mon[104859]: pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:29.618 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:29 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:13:29.618 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:29 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:13:29.618 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:29 smithi032 ceph-mon[104859]: pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:29.619 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:29 smithi032 ceph-mon[104859]: pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:29.619 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:29 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:13:30.711 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:13:31.096 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:13:31.390 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:31 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3599443736' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:31.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:31 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3599443736' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:31.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:31 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3599443736' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:31.484 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:30 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:13:31.484 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:30 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Scheduled restart job, restart counter is at 7. 2023-07-13T02:13:31.484 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:30 smithi191 systemd[1]: Stopped Ceph osd.6 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:31.485 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:30 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Start request repeated too quickly. 2023-07-13T02:13:31.485 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:30 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service: Failed with result 'exit-code'. 2023-07-13T02:13:31.485 INFO:journalctl@ceph.osd.6.smithi191.stdout:Jul 13 02:13:30 smithi191 systemd[1]: Failed to start Ceph osd.6 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:32.097 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:13:32.342 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:13:32.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:32 smithi032 ceph-mon[104859]: pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:32.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:32 smithi032 ceph-mon[116810]: pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:32.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:32 smithi191 ceph-mon[108588]: pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:33.434 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:13:33.812 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:13:34.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:34 smithi032 ceph-mon[116810]: pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:34.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:34 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/796362848' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:34.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:34 smithi032 ceph-mon[104859]: pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:34.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:34 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/796362848' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:34.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:34 smithi191 ceph-mon[108588]: pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:34.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:34 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/796362848' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:34.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:13:35.056 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:13:36.213 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:13:36.373 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:36 smithi032 ceph-mon[116810]: pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:36.374 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:36 smithi032 ceph-mon[104859]: pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:36.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:36 smithi191 ceph-mon[108588]: pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:36.587 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:13:37.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:37 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1293739880' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:37.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:37 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1293739880' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:37.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:37 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1293739880' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:37.484 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:36 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Service RestartSec=10s expired, scheduling restart. 2023-07-13T02:13:37.484 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:36 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Scheduled restart job, restart counter is at 10. 2023-07-13T02:13:37.485 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:36 smithi191 systemd[1]: Stopped Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:37.485 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:36 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Start request repeated too quickly. 2023-07-13T02:13:37.485 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:36 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service: Failed with result 'exit-code'. 2023-07-13T02:13:37.485 INFO:journalctl@ceph.osd.4.smithi191.stdout:Jul 13 02:13:36 smithi191 systemd[1]: Failed to start Ceph osd.4 for 11418e3c-2122-11ee-9b31-001a4aab830c. 2023-07-13T02:13:37.588 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:13:37.835 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:13:38.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:38 smithi032 ceph-mon[104859]: pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:38.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:38 smithi032 ceph-mon[116810]: pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:38.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:38 smithi191 ceph-mon[108588]: pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:39.009 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:13:39.198 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:39 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2160680874' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:39.199 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:39 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2160680874' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:39.404 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:13:39.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:39 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/2160680874' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:40.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:40 smithi032 ceph-mon[104859]: pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:40.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:40 smithi032 ceph-mon[116810]: pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:40.404 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:13:40.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:40 smithi191 ceph-mon[108588]: pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:40.649 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:13:41.765 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:13:42.146 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:13:42.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:42 smithi032 ceph-mon[104859]: pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:42.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:42 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/636216104' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:42.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:42 smithi032 ceph-mon[116810]: pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:42.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:42 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/636216104' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:42.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:42 smithi191 ceph-mon[108588]: pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:42.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:42 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/636216104' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:43.147 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:13:43.393 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:13:44.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:44 smithi032 ceph-mon[104859]: pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:44.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:44 smithi032 ceph-mon[116810]: pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:44.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:44 smithi191 ceph-mon[108588]: pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:44.494 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:13:44.848 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:13:45.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:45 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/639068566' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:45.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:45 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/639068566' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:45.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:45 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/639068566' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:45.849 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:13:46.094 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:13:46.141 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:46 smithi032 ceph-mon[104859]: pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:46.141 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:46 smithi032 ceph-mon[116810]: pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:46.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:46 smithi191 ceph-mon[108588]: pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:47.219 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:13:47.645 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:13:48.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:48 smithi032 ceph-mon[104859]: pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:48.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:48 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3532047190' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:48.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:48 smithi032 ceph-mon[116810]: pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:48.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:48 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3532047190' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:48.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:48 smithi191 ceph-mon[108588]: pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:48.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:48 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3532047190' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:48.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:13:48.891 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:13:50.006 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:13:50.334 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:50 smithi032 ceph-mon[116810]: pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:50.335 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:50 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/4111537043' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:50.335 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:50 smithi032 ceph-mon[104859]: pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:50.335 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:50 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/4111537043' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:50.385 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:13:50.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:50 smithi191 ceph-mon[108588]: pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:50.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:50 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/4111537043' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:51.386 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:13:51.631 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:13:52.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:52 smithi032 ceph-mon[104859]: pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:52.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:52 smithi032 ceph-mon[116810]: pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:52.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:52 smithi191 ceph-mon[108588]: pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:52.764 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:13:53.154 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:13:53.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:53 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1153321142' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:53.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:53 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1153321142' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:53.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:53 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1153321142' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:54.155 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:13:54.400 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:13:54.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:54 smithi191 ceph-mon[108588]: pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:54.538 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:54 smithi032 ceph-mon[116810]: pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:54.539 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:54 smithi032 ceph-mon[104859]: pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:55.508 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:13:55.885 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:13:56.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:56 smithi191 ceph-mon[108588]: pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:56.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:56 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1717371592' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:56.641 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:56 smithi032 ceph-mon[104859]: pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:56.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:56 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1717371592' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:56.642 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:56 smithi032 ceph-mon[116810]: pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:56.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:56 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1717371592' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:56.886 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:13:57.131 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:13:57.254 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:57 smithi032 ceph-mon[116810]: pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:57.254 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:57 smithi032 ceph-mon[104859]: pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:57.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:57 smithi191 ceph-mon[108588]: pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:58.405 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:13:58.638 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:58 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1722251941' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:58.639 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:58 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1722251941' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:58.734 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:58 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1722251941' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:13:58.829 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:13:59.734 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:13:59 smithi191 ceph-mon[108588]: pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:59.830 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:13:59.866 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:13:59 smithi032 ceph-mon[104859]: pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:13:59.866 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:13:59 smithi032 ceph-mon[116810]: pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:00.077 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:14:01.223 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:14:01.610 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:14:02.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:02 smithi032 ceph-mon[104859]: pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:02.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:02 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2077753087' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:02.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:02 smithi032 ceph-mon[116810]: pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:02.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:02 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2077753087' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:02.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:02 smithi191 ceph-mon[108588]: pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:02.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:02 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/2077753087' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:02.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:14:02.856 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:14:03.979 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:14:04.249 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:04 smithi032 ceph-mon[116810]: pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:04.249 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:04 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3760320583' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:04.249 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:04 smithi032 ceph-mon[104859]: pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:04.250 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:04 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3760320583' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:04.367 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:14:04.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:04 smithi191 ceph-mon[108588]: pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:04.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:04 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3760320583' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:05.369 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:14:05.614 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:14:06.371 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:06 smithi032 ceph-mon[116810]: pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:06.371 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:06 smithi032 ceph-mon[104859]: pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:06.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:06 smithi191 ceph-mon[108588]: pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:06.729 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:14:07.118 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:14:07.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:07 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/157072630' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:07.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:07 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/157072630' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:07.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:07 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/157072630' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:08.120 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:14:08.365 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:14:08.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:08 smithi032 ceph-mon[104859]: pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:08.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:08 smithi032 ceph-mon[116810]: pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:08.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:08 smithi191 ceph-mon[108588]: pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:09.492 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:14:09.858 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:14:10.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:10 smithi032 ceph-mon[104859]: pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:10.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:10 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1896255378' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:10.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:10 smithi032 ceph-mon[116810]: pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:10.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:10 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1896255378' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:10.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:10 smithi191 ceph-mon[108588]: pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:10.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:10 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1896255378' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:10.859 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:14:11.104 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:14:12.313 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:14:12.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:12 smithi032 ceph-mon[104859]: pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:12.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:12 smithi032 ceph-mon[116810]: pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:12.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:12 smithi191 ceph-mon[108588]: pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:12.716 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:14:13.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:13 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1418626086' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:13.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:13 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1418626086' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:13.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:13 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1418626086' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:13.717 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:14:13.961 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:14:14.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:14 smithi032 ceph-mon[104859]: pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:14.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:14 smithi032 ceph-mon[116810]: pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:14.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:14 smithi191 ceph-mon[108588]: pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:15.078 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:14:15.381 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:15 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2004963564' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:15.382 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:15 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2004963564' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:15.457 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:14:15.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:15 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/2004963564' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:16.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:16 smithi032 ceph-mon[104859]: pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:16.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:16 smithi032 ceph-mon[116810]: pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:16.458 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:14:16.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:16 smithi191 ceph-mon[108588]: pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:16.703 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:14:17.821 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:14:18.214 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:14:18.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:18 smithi032 ceph-mon[104859]: pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:18.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:18 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1910524440' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:18.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:18 smithi032 ceph-mon[116810]: pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:18.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:18 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1910524440' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:18.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:18 smithi191 ceph-mon[108588]: pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:18.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:18 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1910524440' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:19.216 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:14:19.463 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:14:20.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:20 smithi032 ceph-mon[104859]: pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:20.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:20 smithi032 ceph-mon[116810]: pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:20.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:20 smithi191 ceph-mon[108588]: pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:14:21.065 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:14:21.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:21 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/2002379682' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:21.641 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:21 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2002379682' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:21.642 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:21 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2002379682' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:22.066 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:14:22.312 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:14:22.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:22 smithi032 ceph-mon[104859]: pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:22.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:22 smithi032 ceph-mon[116810]: pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:22.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:22 smithi191 ceph-mon[108588]: pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:23.425 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:14:23.850 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:14:24.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:24 smithi191 ceph-mon[108588]: pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:24.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:24 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3438215214' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:24.641 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:24 smithi032 ceph-mon[104859]: pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:24.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:24 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3438215214' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:24.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:24 smithi032 ceph-mon[116810]: pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:24.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:24 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3438215214' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:24.852 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:14:25.097 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:14:26.243 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:14:26.426 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:26 smithi032 ceph-mon[104859]: pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:26.426 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:26 smithi032 ceph-mon[116810]: pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:26.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:26 smithi191 ceph-mon[108588]: pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:26.625 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:14:27.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:27 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/4247136912' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:27.626 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:14:27.641 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:27 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/4247136912' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:27.642 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:27 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/4247136912' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:27.869 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:14:28.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:28 smithi032 ceph-mon[104859]: pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:28.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:28 smithi032 ceph-mon[116810]: pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:28.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:28 smithi191 ceph-mon[108588]: pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:29.082 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:14:29.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:29 smithi032 ceph-mon[104859]: pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:29.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:29 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:14:29.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:29 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:14:29.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:29 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:14:29.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:29 smithi032 ceph-mon[104859]: pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:29.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:29 smithi032 ceph-mon[104859]: pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:29.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:29 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:14:29.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:29 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/4289830814' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:29.261 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:29 smithi032 ceph-mon[116810]: pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:29.261 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:29 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:14:29.261 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:29 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:14:29.261 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:29 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:14:29.262 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:29 smithi032 ceph-mon[116810]: pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:29.262 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:29 smithi032 ceph-mon[116810]: pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:29.262 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:29 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:14:29.262 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:29 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/4289830814' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:29.457 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:14:29.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:29 smithi191 ceph-mon[108588]: pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:29.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:29 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:14:29.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:29 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:14:29.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:29 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:14:29.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:29 smithi191 ceph-mon[108588]: pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:29.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:29 smithi191 ceph-mon[108588]: pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:29.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:29 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:14:29.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:29 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/4289830814' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:30.458 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:14:30.703 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:14:31.823 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:14:32.230 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:14:32.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:32 smithi032 ceph-mon[104859]: pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:32.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:32 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1581498159' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:32.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:32 smithi032 ceph-mon[116810]: pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:32.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:32 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1581498159' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:32.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:32 smithi191 ceph-mon[108588]: pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:32.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:32 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1581498159' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:33.230 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:14:33.474 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:14:34.299 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:34 smithi032 ceph-mon[104859]: pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:34.300 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:34 smithi032 ceph-mon[116810]: pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:34.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:34 smithi191 ceph-mon[108588]: pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:34.642 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:14:35.031 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:14:35.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:35 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/657169942' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:35.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:35 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/657169942' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:35.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:35 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/657169942' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:36.032 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:14:36.280 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:14:36.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:36 smithi032 ceph-mon[104859]: pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:36.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:36 smithi032 ceph-mon[116810]: pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:36.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:36 smithi191 ceph-mon[108588]: pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:37.441 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:14:37.821 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:14:38.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:38 smithi032 ceph-mon[104859]: pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:38.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:38 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1387971829' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:38.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:38 smithi032 ceph-mon[116810]: pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:38.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:38 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1387971829' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:38.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:38 smithi191 ceph-mon[108588]: pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:38.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:38 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1387971829' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:38.823 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:14:39.067 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:14:40.244 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:14:40.391 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:40 smithi032 ceph-mon[116810]: pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:40.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:40 smithi032 ceph-mon[104859]: pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:40.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:40 smithi191 ceph-mon[108588]: pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:40.638 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:14:41.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:41 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3424961269' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:41.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:41 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3424961269' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:41.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:41 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3424961269' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:41.639 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:14:41.882 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:14:42.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:42 smithi032 ceph-mon[104859]: pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:42.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:42 smithi032 ceph-mon[116810]: pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:42.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:42 smithi191 ceph-mon[108588]: pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:14:43.270 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:43 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/801147804' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:43.270 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:43 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/801147804' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:43.321 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:14:43.484 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:43 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/801147804' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:44.322 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:14:44.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:44 smithi032 ceph-mon[104859]: pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:44.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:44 smithi032 ceph-mon[116810]: pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:44.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:44 smithi191 ceph-mon[108588]: pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:44.567 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:14:45.664 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:14:46.038 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:14:46.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:46 smithi032 ceph-mon[104859]: pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:46.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:46 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2171996037' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:46.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:46 smithi032 ceph-mon[116810]: pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:46.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:46 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2171996037' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:46.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:46 smithi191 ceph-mon[108588]: pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:46.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:46 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/2171996037' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:47.039 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:14:47.285 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:14:48.444 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:14:48.472 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:48 smithi032 ceph-mon[116810]: pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:48.472 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:48 smithi032 ceph-mon[104859]: pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:48.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:48 smithi191 ceph-mon[108588]: pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:48.836 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:14:49.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:49 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3512118966' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:49.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:49 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3512118966' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:49.642 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:49 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3512118966' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:49.837 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:14:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:14:50.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:50 smithi191 ceph-mon[108588]: pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:50.552 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:50 smithi032 ceph-mon[116810]: pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:50.553 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:50 smithi032 ceph-mon[104859]: pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:51.226 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:14:51.586 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:14:52.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:52 smithi191 ceph-mon[108588]: pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:52.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:52 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3198033659' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:52.587 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:14:52.622 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:52 smithi032 ceph-mon[116810]: pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:52.622 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:52 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3198033659' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:52.623 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:52 smithi032 ceph-mon[104859]: pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:52.623 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:52 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3198033659' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:52.833 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:14:53.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:53 smithi032 ceph-mon[104859]: pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:53.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:53 smithi032 ceph-mon[116810]: pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:53.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:53 smithi191 ceph-mon[108588]: pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:54.020 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:14:54.419 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:14:54.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:54 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1528468779' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:54.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:54 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1528468779' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:54.642 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:54 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1528468779' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:55.421 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:14:55.668 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:14:56.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:56 smithi032 ceph-mon[104859]: pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:56.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:56 smithi032 ceph-mon[116810]: pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:56.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:56 smithi191 ceph-mon[108588]: pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:56.801 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:14:57.193 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:14:57.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:57 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/4126881818' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:57.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:57 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/4126881818' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:57.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:57 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/4126881818' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:14:58.194 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:14:58.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:14:58 smithi032 ceph-mon[104859]: pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:58.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:14:58 smithi032 ceph-mon[116810]: pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:58.439 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:14:58.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:14:58 smithi191 ceph-mon[108588]: pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:14:59.602 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:14:59.930 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:15:00.142 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:00 smithi032 ceph-mon[116810]: pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:00.142 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:00 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1091265039' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:00.142 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:00 smithi032 ceph-mon[104859]: pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:00.142 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:00 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1091265039' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:00.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:00 smithi191 ceph-mon[108588]: pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:00.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:00 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1091265039' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:00.932 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:15:01.175 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:15:02.306 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:15:02.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:02 smithi032 ceph-mon[104859]: pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:02.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:02 smithi032 ceph-mon[116810]: pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:02.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:02 smithi191 ceph-mon[108588]: pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:02.741 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:15:03.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:03 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1019881410' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:03.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:03 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1019881410' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:03.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:03 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1019881410' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:03.742 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:15:03.986 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:15:04.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:04 smithi032 ceph-mon[104859]: pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:04.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:04 smithi032 ceph-mon[116810]: pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:04.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:04 smithi191 ceph-mon[108588]: pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:05.156 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:15:05.535 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:15:06.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:06 smithi032 ceph-mon[116810]: pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:06.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:06 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3681074479' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:06.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:06 smithi032 ceph-mon[104859]: pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:06.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:06 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3681074479' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:06.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:06 smithi191 ceph-mon[108588]: pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:06.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:06 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3681074479' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:06.536 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:15:06.780 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:15:07.949 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:15:08.317 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:15:08.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:08 smithi191 ceph-mon[108588]: pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:08.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:08 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/2494379943' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:08.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:08 smithi032 ceph-mon[104859]: pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:08.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:08 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2494379943' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:08.642 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:08 smithi032 ceph-mon[116810]: pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:08.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:08 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2494379943' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:09.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:15:09.564 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:15:10.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:10 smithi191 ceph-mon[108588]: pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:10.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:10 smithi032 ceph-mon[104859]: pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:10.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:10 smithi032 ceph-mon[116810]: pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:15:11.127 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:15:11.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:11 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2405153037' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:11.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:11 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2405153037' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:11.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:11 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/2405153037' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:12.128 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:15:12.377 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:15:12.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:12 smithi191 ceph-mon[108588]: pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:12.495 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:12 smithi032 ceph-mon[116810]: pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:12.496 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:12 smithi032 ceph-mon[104859]: pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:13.487 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:15:13.852 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:15:14.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:14 smithi191 ceph-mon[108588]: pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:14.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:14 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/2186781851' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:14.642 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:14 smithi032 ceph-mon[116810]: pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:14.642 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:14 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2186781851' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:14.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:14 smithi032 ceph-mon[104859]: pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:14.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:14 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2186781851' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:14.853 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:15:15.097 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:15:16.230 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:15:16.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:16 smithi191 ceph-mon[108588]: pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:16.532 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:16 smithi032 ceph-mon[116810]: pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:16.533 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:16 smithi032 ceph-mon[104859]: pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:16.609 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:15:17.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:17 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1673753203' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:17.610 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:15:17.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:17 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1673753203' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:17.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:17 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1673753203' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:17.855 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:15:18.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:18 smithi032 ceph-mon[104859]: pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:18.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:18 smithi032 ceph-mon[116810]: pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:18.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:18 smithi191 ceph-mon[108588]: pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:19.007 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:15:19.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:19 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1909821347' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:19.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:19 smithi032 ceph-mon[104859]: pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:19.221 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:19 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1909821347' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:19.221 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:19 smithi032 ceph-mon[116810]: pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:19.411 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:15:19.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:19 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1909821347' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:19.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:19 smithi191 ceph-mon[108588]: pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:20.412 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:15:20.656 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:15:21.797 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:15:22.175 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:15:22.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:22 smithi032 ceph-mon[104859]: pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:22.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:22 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2261361429' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:22.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:22 smithi032 ceph-mon[116810]: pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:22.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:22 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2261361429' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:22.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:22 smithi191 ceph-mon[108588]: pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:22.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:22 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/2261361429' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:23.176 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:15:23.417 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:15:24.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:24 smithi032 ceph-mon[104859]: pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:24.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:24 smithi032 ceph-mon[116810]: pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:24.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:24 smithi191 ceph-mon[108588]: pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:24.493 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:15:24.907 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:15:25.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:25 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/4218481155' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:25.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:25 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/4218481155' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:25.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:25 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/4218481155' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:25.909 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:15:26.155 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:15:26.252 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:26 smithi032 ceph-mon[116810]: pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:26.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:26 smithi032 ceph-mon[104859]: pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:26.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:26 smithi191 ceph-mon[108588]: pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:27.257 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:15:27.609 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:15:28.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:28 smithi032 ceph-mon[104859]: pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:28.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:28 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2133062877' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:28.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:28 smithi032 ceph-mon[116810]: pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:28.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:28 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2133062877' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:28.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:28 smithi191 ceph-mon[108588]: pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:28.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:28 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/2133062877' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:28.610 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:15:28.852 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:15:30.040 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:15:30.320 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:30 smithi032 ceph-mon[104859]: pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:30.320 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:30 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:15:30.320 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:30 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:15:30.321 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:30 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:15:30.321 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:30 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:15:30.321 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:30 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3053999139' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:30.322 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:30 smithi032 ceph-mon[116810]: pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:30.322 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:30 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:15:30.322 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:30 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:15:30.322 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:30 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:15:30.322 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:30 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:15:30.322 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:30 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3053999139' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:30.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:30 smithi191 ceph-mon[108588]: pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:30.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:30 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:15:30.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:30 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:15:30.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:30 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:15:30.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:30 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:15:30.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:30 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3053999139' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:30.533 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:15:31.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:31 smithi191 ceph-mon[108588]: pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:31.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:31 smithi191 ceph-mon[108588]: pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:31.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:15:31.569 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:31 smithi032 ceph-mon[104859]: pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:31.572 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:31 smithi032 ceph-mon[104859]: pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:31.573 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:31 smithi032 ceph-mon[116810]: pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:31.573 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:31 smithi032 ceph-mon[116810]: pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:31.780 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:15:32.905 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:15:33.342 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:15:33.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:33 smithi191 ceph-mon[108588]: pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:33.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:33 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/9936416' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:33.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:33 smithi032 ceph-mon[104859]: pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:33.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:33 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/9936416' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:33.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:33 smithi032 ceph-mon[116810]: pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:33.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:33 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/9936416' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:34.344 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:15:34.588 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:15:35.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:35 smithi032 ceph-mon[116810]: pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:35.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:35 smithi032 ceph-mon[104859]: pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:35.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:35 smithi191 ceph-mon[108588]: pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:35.713 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:15:36.099 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:15:36.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:36 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3167465124' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:36.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:36 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3167465124' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:36.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:36 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3167465124' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:37.100 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:15:37.347 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:15:37.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:37 smithi032 ceph-mon[104859]: pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:37.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:37 smithi032 ceph-mon[116810]: pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:37.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:37 smithi191 ceph-mon[108588]: pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:38.447 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:38 smithi032 ceph-mon[116810]: pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:38.447 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:38 smithi032 ceph-mon[104859]: pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:38.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:38 smithi191 ceph-mon[108588]: pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:38.509 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:15:38.899 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:15:39.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:39 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3091939670' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:39.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:39 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3091939670' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:39.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:39 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3091939670' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:39.900 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:15:40.144 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:15:41.324 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:15:41.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:41 smithi032 ceph-mon[104859]: pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:41.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:41 smithi032 ceph-mon[116810]: pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:41.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:41 smithi191 ceph-mon[108588]: pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:41.722 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:15:42.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:42 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/661805179' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:42.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:42 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/661805179' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:42.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:42 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/661805179' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:42.723 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:15:42.970 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:15:43.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:43 smithi032 ceph-mon[104859]: pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:43.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:43 smithi032 ceph-mon[116810]: pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:43.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:43 smithi191 ceph-mon[108588]: pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:44.084 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:15:44.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:44 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/4195061994' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:44.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:44 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/4195061994' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:44.472 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:15:44.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:44 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/4195061994' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:45.473 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:15:45.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:45 smithi191 ceph-mon[108588]: pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:45.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:45 smithi032 ceph-mon[116810]: pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:45.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:45 smithi032 ceph-mon[104859]: pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:45.719 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:15:46.885 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:15:47.248 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:15:47.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:47 smithi032 ceph-mon[116810]: pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:47.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1471999568' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:47.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:47 smithi032 ceph-mon[104859]: pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:47.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:47 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1471999568' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:47.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:47 smithi191 ceph-mon[108588]: pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:47.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:47 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1471999568' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:48.249 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:15:48.492 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:15:49.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:49 smithi032 ceph-mon[104859]: pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:49.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:49 smithi032 ceph-mon[116810]: pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:49.485 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:49 smithi191 ceph-mon[108588]: pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:49.658 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:15:50.047 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:15:50.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:50 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1245915206' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:50.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:50 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1245915206' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:50.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:50 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1245915206' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:51.049 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:15:51.296 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:15:51.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:51 smithi032 ceph-mon[116810]: pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:51.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:51 smithi032 ceph-mon[104859]: pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:51.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:51 smithi191 ceph-mon[108588]: pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:52.471 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:15:52.838 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:15:53.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:53 smithi191 ceph-mon[108588]: pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:53.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:53 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/360395899' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:53.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:53 smithi032 ceph-mon[104859]: pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:53.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:53 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/360395899' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:53.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:53 smithi032 ceph-mon[116810]: pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:53.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:53 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/360395899' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:53.838 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:15:54.084 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:15:55.253 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:15:55.425 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:55 smithi032 ceph-mon[104859]: pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:55.425 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:55 smithi032 ceph-mon[116810]: pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:55.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:55 smithi191 ceph-mon[108588]: pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:55.628 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:15:56.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:56 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1353075957' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:56.630 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:15:56.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:56 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1353075957' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:56.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:56 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1353075957' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:56.877 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:15:57.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:57 smithi032 ceph-mon[104859]: pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:57.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:57 smithi032 ceph-mon[116810]: pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:57.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:57 smithi191 ceph-mon[108588]: pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:58.059 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:15:58.370 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:58 smithi032 ceph-mon[104859]: pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:58.370 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:15:58 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/743835215' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:58.370 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:58 smithi032 ceph-mon[116810]: pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:58.371 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:15:58 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/743835215' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:58.436 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:15:58.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:58 smithi191 ceph-mon[108588]: pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:15:58.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:15:58 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/743835215' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:15:59.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:15:59.682 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:16:00.836 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:16:01.222 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:16:01.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:01 smithi032 ceph-mon[104859]: pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:01.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:01 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2452022950' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:01.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:01 smithi032 ceph-mon[116810]: pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:01.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:01 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2452022950' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:01.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:01 smithi191 ceph-mon[108588]: pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:01.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:01 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/2452022950' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:02.223 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:16:02.471 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:16:03.233 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:03 smithi032 ceph-mon[116810]: pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:03.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:03 smithi032 ceph-mon[104859]: pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:03.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:03 smithi191 ceph-mon[108588]: pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:03.614 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:16:04.020 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:16:04.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:04 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1915628109' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:04.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:04 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1915628109' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:04.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:04 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1915628109' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:05.021 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:16:05.269 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:16:05.392 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:05 smithi032 ceph-mon[116810]: pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:05.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:05 smithi032 ceph-mon[104859]: pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:05.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:05 smithi191 ceph-mon[108588]: pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:06.463 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:16:06.853 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:16:07.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:07 smithi191 ceph-mon[108588]: pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:07.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:07 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3727878469' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:07.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:07 smithi032 ceph-mon[104859]: pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:07.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:07 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3727878469' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:07.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:07 smithi032 ceph-mon[116810]: pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:07.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:07 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3727878469' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:07.855 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:16:08.104 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:16:09.265 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:16:09.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:09 smithi191 ceph-mon[108588]: pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:09.490 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:09 smithi032 ceph-mon[104859]: pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:09.490 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:09 smithi032 ceph-mon[116810]: pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:09.663 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:16:10.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:10 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1131175793' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:10.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:10 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1131175793' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:10.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:10 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1131175793' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:10.665 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:16:10.909 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:16:11.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:11 smithi032 ceph-mon[104859]: pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:11.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:11 smithi032 ceph-mon[116810]: pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:11.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:11 smithi191 ceph-mon[108588]: pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:12.049 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:16:12.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:12 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1871055255' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:12.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:12 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1871055255' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:12.446 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:16:12.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:12 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1871055255' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:13.447 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:16:13.482 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:13 smithi032 ceph-mon[104859]: pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:13.483 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:13 smithi032 ceph-mon[116810]: pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:13.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:13 smithi191 ceph-mon[108588]: pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:13.694 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:16:14.838 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:16:15.201 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:16:15.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:15 smithi032 ceph-mon[104859]: pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:15.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:15 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3503555033' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:15.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:15 smithi032 ceph-mon[116810]: pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:15.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:15 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3503555033' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:15.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:15 smithi191 ceph-mon[108588]: pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:15.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:15 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3503555033' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:16.203 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:16:16.449 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:16:17.190 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:17 smithi032 ceph-mon[116810]: pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:17.190 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:17 smithi032 ceph-mon[104859]: pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:17.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:17 smithi191 ceph-mon[108588]: pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:17.567 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:16:17.951 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:16:18.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:18 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/2355828186' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:18.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:18 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2355828186' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:18.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:18 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2355828186' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:18.952 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:16:19.199 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:16:19.330 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:19 smithi032 ceph-mon[116810]: pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:19.331 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:19 smithi032 ceph-mon[104859]: pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:19.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:19 smithi191 ceph-mon[108588]: pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:20.369 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:16:20.778 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:16:21.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:21 smithi191 ceph-mon[108588]: pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:21.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:21 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/611343393' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:21.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:21 smithi032 ceph-mon[104859]: pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:21.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:21 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/611343393' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:21.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:21 smithi032 ceph-mon[116810]: pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:21.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:21 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/611343393' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:21.780 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:16:22.025 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:16:23.156 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:16:23.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:23 smithi032 ceph-mon[104859]: pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:23.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:23 smithi032 ceph-mon[116810]: pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:23.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:23 smithi191 ceph-mon[108588]: pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:23.534 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:16:24.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:24 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3617536076' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:24.536 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:16:24.570 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:24 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3617536076' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:24.570 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:24 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3617536076' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:24.780 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:16:25.152 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:25 smithi032 ceph-mon[104859]: pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:25.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:25 smithi191 ceph-mon[108588]: pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:25.532 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:25 smithi032 ceph-mon[116810]: pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:25.913 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:16:26.283 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:16:26.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:26 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/777044952' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:26.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:26 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/777044952' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:26.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:26 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/777044952' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:27.284 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:16:27.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:27 smithi191 ceph-mon[108588]: pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:27.531 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:16:27.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:27 smithi032 ceph-mon[104859]: pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:27.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:27 smithi032 ceph-mon[116810]: pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:28.686 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:16:29.084 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:16:29.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:29 smithi032 ceph-mon[104859]: pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:29.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:29 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1259696735' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:29.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:29 smithi032 ceph-mon[116810]: pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:29.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:29 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1259696735' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:29.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:29 smithi191 ceph-mon[108588]: pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:29.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:29 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1259696735' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:30.085 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:16:30.326 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:16:30.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:30 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:16:30.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:30 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:16:30.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:30 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:16:31.442 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:31 smithi032 ceph-mon[116810]: pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:31.442 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:31 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:16:31.442 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:31 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:16:31.442 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:31 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:16:31.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:31 smithi032 ceph-mon[104859]: pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:31.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:31 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:16:31.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:31 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:16:31.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:31 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:16:31.449 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:16:31.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:31 smithi191 ceph-mon[108588]: pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:31.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:31 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:16:31.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:31 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:16:31.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:31 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:16:32.018 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:16:32.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:32 smithi191 ceph-mon[108588]: pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:32.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:32 smithi191 ceph-mon[108588]: pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:32.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:32 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/919480005' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:32.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:32 smithi032 ceph-mon[104859]: pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:32.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:32 smithi032 ceph-mon[104859]: pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:32.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:32 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/919480005' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:32.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:32 smithi032 ceph-mon[116810]: pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:32.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:32 smithi032 ceph-mon[116810]: pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:32.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:32 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/919480005' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:33.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:16:33.264 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:16:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:16:34.425 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:34 smithi032 ceph-mon[116810]: pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:34.426 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:34 smithi032 ceph-mon[104859]: pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:34.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:34 smithi191 ceph-mon[108588]: pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:34.766 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:16:35.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:35 smithi191 ceph-mon[108588]: pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:35.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:35 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1043000561' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:35.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:35 smithi032 ceph-mon[104859]: pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:35.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:35 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1043000561' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:35.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:35 smithi032 ceph-mon[116810]: pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:35.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:35 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1043000561' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:35.768 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:16:36.013 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:16:37.121 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:16:37.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:37 smithi032 ceph-mon[104859]: pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:37.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:37 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1753125777' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:37.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:37 smithi032 ceph-mon[116810]: pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:37.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:37 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1753125777' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:37.507 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:16:37.736 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:37 smithi191 ceph-mon[108588]: pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:37.736 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:37 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1753125777' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:38.508 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:16:38.752 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:16:39.529 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:39 smithi032 ceph-mon[104859]: pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:39.530 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:39 smithi032 ceph-mon[116810]: pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:39.736 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:39 smithi191 ceph-mon[108588]: pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:39.886 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:16:40.261 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:16:40.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:40 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2907439463' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:40.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:40 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2907439463' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:40.736 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:40 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/2907439463' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:41.263 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:16:41.509 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:16:42.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:42 smithi032 ceph-mon[104859]: pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:42.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:42 smithi032 ceph-mon[116810]: pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:42.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:42 smithi191 ceph-mon[108588]: pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:42.661 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:16:43.091 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:16:43.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:43 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/4088934620' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:43.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:43 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/4088934620' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:43.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:43 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/4088934620' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:44.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:16:44.336 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:16:44.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:44 smithi032 ceph-mon[104859]: pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:44.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:44 smithi032 ceph-mon[116810]: pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:44.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:44 smithi191 ceph-mon[108588]: pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:45.459 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:16:45.840 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:16:46.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:46 smithi191 ceph-mon[108588]: pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:46.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:46 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3495993040' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:46.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:46 smithi032 ceph-mon[104859]: pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:46.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:46 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3495993040' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:46.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:46 smithi032 ceph-mon[116810]: pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:46.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:46 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3495993040' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:46.841 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:16:47.087 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:16:48.321 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:16:48.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:48 smithi191 ceph-mon[108588]: pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:48.552 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:48 smithi032 ceph-mon[116810]: pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:48.552 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:48 smithi032 ceph-mon[104859]: pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:48.708 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:16:49.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:49 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/135337702' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:49.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:49 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/135337702' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:49.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:49 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/135337702' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:49.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:16:49.953 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:16:50.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:50 smithi032 ceph-mon[104859]: pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:50.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:50 smithi032 ceph-mon[116810]: pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:50.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:50 smithi191 ceph-mon[108588]: pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:51.038 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:16:51.363 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:51 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1115634988' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:51.364 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:51 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1115634988' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:51.439 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:16:51.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:51 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1115634988' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:52.440 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:16:52.475 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:52 smithi032 ceph-mon[104859]: pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:52.475 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:52 smithi032 ceph-mon[116810]: pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:52.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:52 smithi191 ceph-mon[108588]: pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:52.686 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:16:53.785 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:16:54.164 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:16:54.393 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:54 smithi032 ceph-mon[116810]: pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:54.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:54 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1934691289' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:54.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:54 smithi032 ceph-mon[104859]: pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:54.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:54 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1934691289' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:54.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:54 smithi191 ceph-mon[108588]: pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:54.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:54 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1934691289' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:55.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:16:55.413 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:16:56.176 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:56 smithi032 ceph-mon[104859]: pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:56.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:56 smithi191 ceph-mon[108588]: pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:56.496 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:16:56.496 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:56 smithi032 ceph-mon[116810]: pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:56.889 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:16:57.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:57 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/471788597' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:57.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:57 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/471788597' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:57.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:57 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/471788597' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:16:57.889 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:16:58.137 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:16:58.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:58 smithi191 ceph-mon[108588]: pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:58.629 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:58 smithi032 ceph-mon[116810]: pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:58.630 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:58 smithi032 ceph-mon[104859]: pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:59.243 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:16:59.434 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:16:59 smithi032 ceph-mon[116810]: pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:59.435 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:16:59 smithi032 ceph-mon[104859]: pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:59.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:16:59 smithi191 ceph-mon[108588]: pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:16:59.621 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:17:00.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:00 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/2833457072' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:00.623 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:17:00.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:00 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2833457072' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:00.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:00 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2833457072' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:00.868 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:17:01.370 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:01 smithi032 ceph-mon[116810]: pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:01.371 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:01 smithi032 ceph-mon[104859]: pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:01.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:01 smithi191 ceph-mon[108588]: pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:01.973 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:17:02.293 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:02 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3198528389' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:02.294 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:02 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3198528389' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:02.355 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:17:02.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:02 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3198528389' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:03.356 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:17:03.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:03 smithi191 ceph-mon[108588]: pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:03.602 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:17:03.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:03 smithi032 ceph-mon[104859]: pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:03.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:03 smithi032 ceph-mon[116810]: pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:04.781 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:17:05.180 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:17:05.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:05 smithi032 ceph-mon[116810]: pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:05.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:05 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2587484517' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:05.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:05 smithi032 ceph-mon[104859]: pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:05.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:05 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2587484517' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:05.737 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:05 smithi191 ceph-mon[108588]: pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:05.737 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:05 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/2587484517' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:06.181 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:17:06.427 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:17:07.303 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:07 smithi032 ceph-mon[104859]: pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:07.560 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:17:07.582 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:07 smithi032 ceph-mon[116810]: pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:07.736 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:07 smithi191 ceph-mon[108588]: pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:07.945 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:17:08.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:08 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1639298767' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:08.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:08 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1639298767' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:08.736 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:08 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1639298767' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:08.946 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:17:09.193 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:17:09.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:09 smithi032 ceph-mon[104859]: pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:09.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:09 smithi032 ceph-mon[116810]: pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:09.736 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:09 smithi191 ceph-mon[108588]: pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:10.337 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:17:10.747 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:17:11.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:11 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1236240677' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:11.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:11 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1236240677' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:11.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:11 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1236240677' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:11.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:17:11.992 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:17:12.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:12 smithi032 ceph-mon[104859]: pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:12.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:12 smithi032 ceph-mon[116810]: pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:12.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:12 smithi191 ceph-mon[108588]: pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:13.164 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:17:13.545 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:17:14.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:14 smithi191 ceph-mon[108588]: pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:14.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:14 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3394211111' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:14.546 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:17:14.581 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:14 smithi032 ceph-mon[104859]: pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:14.582 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:14 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3394211111' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:14.582 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:14 smithi032 ceph-mon[116810]: pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:14.582 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:14 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3394211111' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:14.792 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:17:15.882 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:17:16.262 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:17:16.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:16 smithi191 ceph-mon[108588]: pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:16.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:16 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1814204987' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:16.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:16 smithi032 ceph-mon[116810]: pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:16.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:16 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1814204987' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:16.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:16 smithi032 ceph-mon[104859]: pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:16.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:16 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1814204987' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:17.263 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:17:17.510 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:17:18.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:18 smithi032 ceph-mon[116810]: pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:18.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:18 smithi032 ceph-mon[104859]: pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:18.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:18 smithi191 ceph-mon[108588]: pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:18.597 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:17:18.978 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:17:19.486 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:19 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/4057415261' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:19.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:19 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/4057415261' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:19.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:19 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/4057415261' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:19.979 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:17:20.226 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:17:20.347 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:20 smithi032 ceph-mon[116810]: pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:20.348 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:20 smithi032 ceph-mon[104859]: pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:20.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:20 smithi191 ceph-mon[108588]: pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:21.356 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:17:21.719 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:17:22.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:22 smithi191 ceph-mon[108588]: pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:22.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:22 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/2680881317' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:22.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:22 smithi032 ceph-mon[104859]: pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:22.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:22 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/2680881317' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:22.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:22 smithi032 ceph-mon[116810]: pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:22.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:22 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/2680881317' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:22.720 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:17:22.966 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:17:24.081 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:17:24.343 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:24 smithi032 ceph-mon[116810]: pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:24.343 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:24 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/619389624' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:24.343 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:24 smithi032 ceph-mon[104859]: pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:24.344 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:24 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/619389624' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:24.469 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:17:24.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:24 smithi191 ceph-mon[108588]: pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:24.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:24 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/619389624' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:25.471 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:17:25.714 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:17:26.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:26 smithi191 ceph-mon[108588]: pgmap v278: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:26.515 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:26 smithi032 ceph-mon[116810]: pgmap v278: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:26.515 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:26 smithi032 ceph-mon[104859]: pgmap v278: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:26.913 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:17:27.302 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:17:27.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:27 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1995882585' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:27.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:27 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1995882585' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:27.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:27 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1995882585' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:28.303 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:17:28.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:28 smithi191 ceph-mon[108588]: pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:28.550 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:17:28.643 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:28 smithi032 ceph-mon[116810]: pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:28.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:28 smithi032 ceph-mon[104859]: pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:29.655 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:17:30.044 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:17:30.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:30 smithi032 ceph-mon[104859]: pgmap v280: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:30.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:30 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3260376208' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:30.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:30 smithi032 ceph-mon[116810]: pgmap v280: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:30.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:30 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3260376208' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:30.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:30 smithi191 ceph-mon[108588]: pgmap v280: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:30.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:30 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3260376208' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:31.045 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:17:31.276 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:17:31.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:31 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:17:31.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:31 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:17:31.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:31 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:17:31.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:31 smithi032 ceph-mon[104859]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:17:31.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:31 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:17:31.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:31 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:17:31.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:31 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:17:31.388 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:31 smithi032 ceph-mon[116810]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:17:31.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:31 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-13T02:17:31.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:31 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-13T02:17:31.488 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:31 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-13T02:17:31.488 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:31 smithi191 ceph-mon[108588]: from='mgr.14152 172.21.15.32:0/588018287' entity='mgr.y' 2023-07-13T02:17:32.423 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:17:32.455 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:32 smithi032 ceph-mon[116810]: pgmap v281: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:32.455 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:32 smithi032 ceph-mon[116810]: pgmap v282: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:32.455 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:32 smithi032 ceph-mon[116810]: pgmap v283: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:32.455 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:32 smithi032 ceph-mon[104859]: pgmap v281: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:32.455 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:32 smithi032 ceph-mon[104859]: pgmap v282: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:32.456 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:32 smithi032 ceph-mon[104859]: pgmap v283: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:32.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:32 smithi191 ceph-mon[108588]: pgmap v281: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:32.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:32 smithi191 ceph-mon[108588]: pgmap v282: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:32.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:32 smithi191 ceph-mon[108588]: pgmap v283: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:32.818 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:17:33.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:33 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1518836730' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:33.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:33 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1518836730' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:33.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:33 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1518836730' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:33.820 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:17:34.066 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:17:34.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:34 smithi191 ceph-mon[108588]: pgmap v284: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:34.642 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:34 smithi032 ceph-mon[116810]: pgmap v284: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:34.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:34 smithi032 ceph-mon[104859]: pgmap v284: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:35.239 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:17:35.617 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:17:36.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:36 smithi191 ceph-mon[108588]: pgmap v285: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:36.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:36 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/780763377' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:36.618 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:17:36.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:36 smithi032 ceph-mon[104859]: pgmap v285: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:36.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:36 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/780763377' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:36.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:36 smithi032 ceph-mon[116810]: pgmap v285: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:36.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:36 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/780763377' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:36.865 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:17:37.958 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:17:38.291 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:38 smithi032 ceph-mon[116810]: pgmap v286: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:38.292 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:38 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/605024766' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:38.292 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:38 smithi032 ceph-mon[104859]: pgmap v286: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:38.292 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:38 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/605024766' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:38.342 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:17:38.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:38 smithi191 ceph-mon[108588]: pgmap v286: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:38.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:38 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/605024766' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:39.343 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:17:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:17:40.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:40 smithi032 ceph-mon[104859]: pgmap v287: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:40.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:40 smithi032 ceph-mon[116810]: pgmap v287: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:40.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:40 smithi191 ceph-mon[108588]: pgmap v287: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:40.693 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:17:41.059 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:17:41.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:41 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1076281272' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:41.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:41 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1076281272' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:41.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:41 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1076281272' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:42.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:17:42.307 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:17:42.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:42 smithi032 ceph-mon[104859]: pgmap v288: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:42.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:42 smithi032 ceph-mon[116810]: pgmap v288: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:42.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:42 smithi191 ceph-mon[108588]: pgmap v288: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:43.467 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:17:44.005 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:17:44.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:44 smithi032 ceph-mon[104859]: pgmap v289: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:44.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:44 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/3547323842' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:44.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:44 smithi032 ceph-mon[116810]: pgmap v289: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:44.395 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:44 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/3547323842' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:44.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:44 smithi191 ceph-mon[108588]: pgmap v289: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:44.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:44 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/3547323842' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:45.006 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:17:45.251 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:17:46.385 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:17:46.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:46 smithi191 ceph-mon[108588]: pgmap v290: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:46.604 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:46 smithi032 ceph-mon[104859]: pgmap v290: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:46.604 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:46 smithi032 ceph-mon[116810]: pgmap v290: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:46.791 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:17:47.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:47 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1703715897' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:47.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:47 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1703715897' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:47.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:47 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1703715897' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:47.792 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:17:48.038 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:17:48.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:48 smithi032 ceph-mon[104859]: pgmap v291: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:48.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:48 smithi032 ceph-mon[116810]: pgmap v291: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:48.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:48 smithi191 ceph-mon[108588]: pgmap v291: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:49.125 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:17:49.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:49 smithi032 ceph-mon[104859]: pgmap v292: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:49.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:49 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/1132692556' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:49.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:49 smithi032 ceph-mon[116810]: pgmap v292: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:49.394 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:49 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/1132692556' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:49.475 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:17:49.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:49 smithi191 ceph-mon[108588]: pgmap v292: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:49.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:49 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/1132692556' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:50.476 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:17:50.721 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:17:51.800 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:17:52.176 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:17:52.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:52 smithi032 ceph-mon[104859]: pgmap v293: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:52.395 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:52 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/906121171' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:52.395 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:52 smithi032 ceph-mon[116810]: pgmap v293: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:52.395 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:52 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/906121171' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:52.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:52 smithi191 ceph-mon[108588]: pgmap v293: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:52.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:52 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/906121171' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:53.178 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:17:53.423 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:17:54.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:54 smithi191 ceph-mon[108588]: pgmap v294: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:54.498 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:54 smithi032 ceph-mon[104859]: pgmap v294: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:54.498 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:54 smithi032 ceph-mon[116810]: pgmap v294: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:54.525 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:17:54.899 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:17:55.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:55 smithi191 ceph-mon[108588]: from='client.? 172.21.15.32:0/347202865' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:55.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:55 smithi032 ceph-mon[104859]: from='client.? 172.21.15.32:0/347202865' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:55.644 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:55 smithi032 ceph-mon[116810]: from='client.? 172.21.15.32:0/347202865' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-13T02:17:55.901 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3990f91ac125621431ef1b77c2fc96421bec9e51 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11418e3c-2122-11ee-9b31-001a4aab830c -- ceph osd stat -f json 2023-07-13T02:17:56.147 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/mon.a/config 2023-07-13T02:17:56.300 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:56 smithi032 ceph-mon[116810]: pgmap v295: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:56.300 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:56 smithi032 ceph-mon[104859]: pgmap v295: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:56.487 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:56 smithi191 ceph-mon[108588]: pgmap v295: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-13T02:17:57.316 INFO:teuthology.orchestra.run.smithi032.stdout: 2023-07-13T02:17:57.724 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1689214331,"num_remapped_pgs":0} 2023-07-13T02:17:57.724 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3990f91ac125621431ef1b77c2fc96421bec9e51/qa/tasks/cephadm.py", line 894, in ceph_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (121) after waiting for 120 seconds 2023-07-13T02:17:57.725 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-13T02:17:57.756 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-13T02:17:57.795 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-07-13T02:17:57.795 DEBUG:teuthology.orchestra.run.smithi032:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-13T02:17:57.815 DEBUG:teuthology.orchestra.run.smithi191:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-13T02:17:57.855 INFO:tasks.cephadm:Stopping all daemons... 2023-07-13T02:17:57.855 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-07-13T02:17:57.856 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mon.a 2023-07-13T02:17:58.144 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:57 smithi032 systemd[1]: Stopping Ceph mon.a for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:17:58.144 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:58 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mon-a[104855]: 2023-07-13T02:17:58.060+0000 7fef8d375700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-13T02:17:58.144 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:58 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mon-a[104855]: 2023-07-13T02:17:58.060+0000 7fef8d375700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-07-13T02:17:58.527 INFO:journalctl@ceph.mon.a.smithi032.stdout:Jul 13 02:17:58 smithi032 bash[171419]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mon-a 2023-07-13T02:17:58.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mon.a.service' 2023-07-13T02:17:58.579 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-13T02:17:58.580 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-07-13T02:17:58.580 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2023-07-13T02:17:58.581 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mon.c 2023-07-13T02:17:58.796 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:58 smithi032 systemd[1]: Stopping Ceph mon.c for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:17:59.144 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:58 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mon-c[116806]: 2023-07-13T02:17:58.785+0000 7f349e78c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-13T02:17:59.144 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:58 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mon-c[116806]: 2023-07-13T02:17:58.785+0000 7f349e78c700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2023-07-13T02:17:59.144 INFO:journalctl@ceph.mon.c.smithi032.stdout:Jul 13 02:17:58 smithi032 bash[171555]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mon-c 2023-07-13T02:17:59.231 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mon.c.service' 2023-07-13T02:17:59.266 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-13T02:17:59.267 INFO:tasks.cephadm.mon.b:Stopped mon.c 2023-07-13T02:17:59.267 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2023-07-13T02:17:59.268 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mon.b 2023-07-13T02:17:59.628 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:59 smithi191 systemd[1]: Stopping Ceph mon.b for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:17:59.628 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:59 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mon-b[108581]: 2023-07-13T02:17:59.487+0000 7f763831d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-13T02:17:59.628 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:59 smithi191 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mon-b[108581]: 2023-07-13T02:17:59.487+0000 7f763831d700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2023-07-13T02:17:59.935 INFO:journalctl@ceph.mon.b.smithi191.stdout:Jul 13 02:17:59 smithi191 bash[141379]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mon-b 2023-07-13T02:17:59.942 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mon.b.service' 2023-07-13T02:17:59.994 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-13T02:17:59.994 INFO:tasks.cephadm.mon.b:Stopped mon.b 2023-07-13T02:17:59.995 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-07-13T02:17:59.995 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mgr.y 2023-07-13T02:18:00.377 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:18:00 smithi032 systemd[1]: Stopping Ceph mgr.y for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:18:00.377 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:18:00 smithi032 ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y[105086]: 2023-07-13T02:18:00.201+0000 7f7dda5a8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-13T02:18:00.644 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Jul 13 02:18:00 smithi032 bash[171690]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-y 2023-07-13T02:18:00.707 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mgr.y.service' 2023-07-13T02:18:00.743 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-13T02:18:00.744 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-07-13T02:18:00.744 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-07-13T02:18:00.745 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mgr.x 2023-07-13T02:18:01.066 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:18:00 smithi191 systemd[1]: Stopping Ceph mgr.x for 11418e3c-2122-11ee-9b31-001a4aab830c... 2023-07-13T02:18:01.343 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:18:01 smithi191 bash[141528]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c-mgr-x 2023-07-13T02:18:01.343 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Jul 13 02:18:01 smithi191 systemd[1]: ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mgr.x.service: Main process exited, code=exited, status=143/n/a 2023-07-13T02:18:01.348 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@mgr.x.service' 2023-07-13T02:18:01.383 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-13T02:18:01.383 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-07-13T02:18:01.384 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-07-13T02:18:01.384 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0 2023-07-13T02:18:01.426 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.0.service' 2023-07-13T02:18:01.506 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-13T02:18:01.506 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-07-13T02:18:01.507 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-07-13T02:18:01.507 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1 2023-07-13T02:18:01.583 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.1.service' 2023-07-13T02:18:01.660 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-13T02:18:01.661 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-07-13T02:18:01.662 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-07-13T02:18:01.662 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2 2023-07-13T02:18:01.736 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.2.service' 2023-07-13T02:18:01.811 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-13T02:18:01.811 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-07-13T02:18:01.812 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-07-13T02:18:01.812 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.3 2023-07-13T02:18:01.886 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.3.service' 2023-07-13T02:18:01.959 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-13T02:18:01.959 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-07-13T02:18:01.960 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-07-13T02:18:01.960 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4 2023-07-13T02:18:01.996 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.4.service' 2023-07-13T02:18:02.070 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-13T02:18:02.070 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-07-13T02:18:02.070 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-07-13T02:18:02.071 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5 2023-07-13T02:18:02.143 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.5.service' 2023-07-13T02:18:02.216 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-13T02:18:02.217 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-07-13T02:18:02.217 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-07-13T02:18:02.217 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6 2023-07-13T02:18:02.288 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.6.service' 2023-07-13T02:18:02.360 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-13T02:18:02.360 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-07-13T02:18:02.361 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-07-13T02:18:02.361 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.7 2023-07-13T02:18:02.432 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-11418e3c-2122-11ee-9b31-001a4aab830c@osd.7.service' 2023-07-13T02:18:02.504 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-13T02:18:02.504 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-07-13T02:18:02.505 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 11418e3c-2122-11ee-9b31-001a4aab830c --force --keep-logs 2023-07-13T02:18:03.893 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 11418e3c-2122-11ee-9b31-001a4aab830c --force --keep-logs 2023-07-13T02:18:05.100 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-13T02:18:05.139 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-13T02:18:05.169 INFO:tasks.cephadm:Archiving crash dumps... 2023-07-13T02:18:05.170 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/crash to /home/teuthworker/archive/gabrioux-2023-07-13_00:38:31-orch:cephadm-cv-refactor-osd-objectstore-distro-default-smithi/7335617/remote/smithi032/crash 2023-07-13T02:18:05.171 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/crash -- . 2023-07-13T02:18:05.222 INFO:teuthology.orchestra.run.smithi032.stderr:tar: /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-13T02:18:05.223 INFO:teuthology.orchestra.run.smithi032.stderr:tar: Error is not recoverable: exiting now 2023-07-13T02:18:05.224 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/crash to /home/teuthworker/archive/gabrioux-2023-07-13_00:38:31-orch:cephadm-cv-refactor-osd-objectstore-distro-default-smithi/7335617/remote/smithi191/crash 2023-07-13T02:18:05.225 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/crash -- . 2023-07-13T02:18:05.261 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-13T02:18:05.261 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now 2023-07-13T02:18:05.262 INFO:tasks.cephadm:Checking cluster log for badness... 2023-07-13T02:18:05.263 DEBUG:teuthology.orchestra.run.smithi032:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2023-07-13T02:18:05.295 INFO:teuthology.orchestra.run.smithi032.stderr:grep: /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph.log: No such file or directory 2023-07-13T02:18:05.296 INFO:tasks.cephadm:Compressing logs... 2023-07-13T02:18:05.297 DEBUG:teuthology.orchestra.run.smithi032:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-13T02:18:05.339 DEBUG:teuthology.orchestra.run.smithi191:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-13T02:18:05.365 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-13T02:18:05.366 INFO:teuthology.orchestra.run.smithi191.stderr:: No such file or directory 2023-07-13T02:18:05.366 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.6.log 2023-07-13T02:18:05.367 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-volume.log 2023-07-13T02:18:05.367 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.7.log 2023-07-13T02:18:05.367 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.6.log: /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-mgr.x.log 2023-07-13T02:18:05.368 INFO:teuthology.orchestra.run.smithi191.stderr: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-13T02:18:05.368 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-mon.b.log 2023-07-13T02:18:05.368 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.4.log 2023-07-13T02:18:05.368 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-mgr.x.log: /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.5.log 2023-07-13T02:18:05.369 INFO:teuthology.orchestra.run.smithi032.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-13T02:18:05.370 INFO:teuthology.orchestra.run.smithi032.stderr:: No such file or directory 2023-07-13T02:18:05.370 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-volume.log 2023-07-13T02:18:05.370 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.2.log 2023-07-13T02:18:05.370 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-mgr.y.log 2023-07-13T02:18:05.370 INFO:teuthology.orchestra.run.smithi032.stderr:gzip /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.2.log: -5 --verbose -- /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.3.log 2023-07-13T02:18:05.370 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.0.log 2023-07-13T02:18:05.371 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.3.log: /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.0.log: 94.7% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-13T02:18:05.371 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-mon.a.log 2023-07-13T02:18:05.372 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-mon.c.log 2023-07-13T02:18:05.374 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.4.log: /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.5.log: 91.8% -- replaced with /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-mgr.x.log.gz 2023-07-13T02:18:05.374 INFO:teuthology.orchestra.run.smithi191.stderr: 94.5% -- replaced with /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.6.log.gz 2023-07-13T02:18:05.375 INFO:teuthology.orchestra.run.smithi191.stderr: 91.9% -- replaced with /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-volume.log.gz 2023-07-13T02:18:05.375 INFO:teuthology.orchestra.run.smithi191.stderr: 94.6% -- replaced with /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.7.log.gz 2023-07-13T02:18:05.376 INFO:teuthology.orchestra.run.smithi191.stderr: 94.6% -- replaced with /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.4.log.gz 2023-07-13T02:18:05.376 INFO:teuthology.orchestra.run.smithi191.stderr: 94.5% -- replaced with /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.5.log.gz 2023-07-13T02:18:05.376 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-mon.a.log: /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-mon.c.log: 94.5% -- replaced with /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.2.log.gz 2023-07-13T02:18:05.376 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.1.log 2023-07-13T02:18:05.377 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.1.log: 92.0% -- replaced with /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-volume.log.gz 2023-07-13T02:18:05.377 INFO:teuthology.orchestra.run.smithi032.stderr: 94.6% -- replaced with /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.3.log.gz 2023-07-13T02:18:05.378 INFO:teuthology.orchestra.run.smithi032.stderr: 94.5% -- replaced with /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.0.log.gz 2023-07-13T02:18:05.381 INFO:teuthology.orchestra.run.smithi032.stderr: 94.5% -- replaced with /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-osd.1.log.gz 2023-07-13T02:18:05.450 INFO:teuthology.orchestra.run.smithi032.stderr: 90.8% -- replaced with /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-mgr.y.log.gz 2023-07-13T02:18:05.459 INFO:teuthology.orchestra.run.smithi191.stderr: 92.2% -- replaced with /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-mon.b.log.gz 2023-07-13T02:18:05.460 INFO:teuthology.orchestra.run.smithi191.stderr: 2023-07-13T02:18:05.460 INFO:teuthology.orchestra.run.smithi191.stderr:real 0m0.105s 2023-07-13T02:18:05.461 INFO:teuthology.orchestra.run.smithi191.stderr:user 0m0.138s 2023-07-13T02:18:05.461 INFO:teuthology.orchestra.run.smithi191.stderr:sys 0m0.020s 2023-07-13T02:18:05.472 INFO:teuthology.orchestra.run.smithi032.stderr: 92.2% -- replaced with /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-mon.c.log.gz 2023-07-13T02:18:05.663 INFO:teuthology.orchestra.run.smithi032.stderr: 91.0% -- replaced with /var/log/ceph/11418e3c-2122-11ee-9b31-001a4aab830c/ceph-mon.a.log.gz 2023-07-13T02:18:05.665 INFO:teuthology.orchestra.run.smithi032.stderr: 2023-07-13T02:18:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:real 0m0.309s 2023-07-13T02:18:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:user 0m0.501s 2023-07-13T02:18:05.666 INFO:teuthology.orchestra.run.smithi032.stderr:sys 0m0.042s 2023-07-13T02:18:05.667 INFO:tasks.cephadm:Archiving logs... 2023-07-13T02:18:05.667 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/log/ceph to /home/teuthworker/archive/gabrioux-2023-07-13_00:38:31-orch:cephadm-cv-refactor-osd-objectstore-distro-default-smithi/7335617/remote/smithi032/log 2023-07-13T02:18:05.669 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-13T02:18:05.814 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/log/ceph to /home/teuthworker/archive/gabrioux-2023-07-13_00:38:31-orch:cephadm-cv-refactor-osd-objectstore-distro-default-smithi/7335617/remote/smithi191/log 2023-07-13T02:18:05.815 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-13T02:18:05.864 INFO:tasks.cephadm:Removing cluster... 2023-07-13T02:18:05.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 11418e3c-2122-11ee-9b31-001a4aab830c --force 2023-07-13T02:18:06.195 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 11418e3c-2122-11ee-9b31-001a4aab830c --force 2023-07-13T02:18:06.522 INFO:tasks.cephadm:Removing cephadm ... 2023-07-13T02:18:06.522 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-13T02:18:06.543 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-13T02:18:06.560 INFO:tasks.cephadm:Teardown complete 2023-07-13T02:18:06.561 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3990f91ac125621431ef1b77c2fc96421bec9e51/qa/tasks/cephadm.py", line 1643, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_3990f91ac125621431ef1b77c2fc96421bec9e51/qa/tasks/cephadm.py", line 1558, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3990f91ac125621431ef1b77c2fc96421bec9e51/qa/tasks/cephadm.py", line 894, in ceph_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (121) after waiting for 120 seconds 2023-07-13T02:18:06.631 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7f82ea8e3a5d4ee49421563f21eb24a7 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3990f91ac125621431ef1b77c2fc96421bec9e51/qa/tasks/cephadm.py", line 1643, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_3990f91ac125621431ef1b77c2fc96421bec9e51/qa/tasks/cephadm.py", line 1558, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3990f91ac125621431ef1b77c2fc96421bec9e51/qa/tasks/cephadm.py", line 894, in ceph_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (121) after waiting for 120 seconds 2023-07-13T02:18:06.634 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-07-13T02:18:06.647 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2023-07-13T02:18:06.675 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi032:/dev/vg_nvme/lv_1... 2023-07-13T02:18:06.676 DEBUG:teuthology.orchestra.run.smithi032:> sudo nvme disconnect -n lv_1 2023-07-13T02:18:06.811 INFO:teuthology.orchestra.run.smithi032.stdout:NQN:lv_1 disconnected 1 controller(s) 2023-07-13T02:18:06.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-13T02:18:06.813 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi032:/dev/vg_nvme/lv_2... 2023-07-13T02:18:06.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo nvme disconnect -n lv_2 2023-07-13T02:18:06.940 INFO:teuthology.orchestra.run.smithi032.stdout:NQN:lv_2 disconnected 1 controller(s) 2023-07-13T02:18:06.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-13T02:18:06.941 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi032:/dev/vg_nvme/lv_3... 2023-07-13T02:18:06.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo nvme disconnect -n lv_3 2023-07-13T02:18:07.063 INFO:teuthology.orchestra.run.smithi032.stdout:NQN:lv_3 disconnected 1 controller(s) 2023-07-13T02:18:07.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-13T02:18:07.065 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi032:/dev/vg_nvme/lv_4... 2023-07-13T02:18:07.065 DEBUG:teuthology.orchestra.run.smithi032:> sudo nvme disconnect -n lv_4 2023-07-13T02:18:07.187 INFO:teuthology.orchestra.run.smithi032.stdout:NQN:lv_4 disconnected 1 controller(s) 2023-07-13T02:18:07.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-13T02:18:07.189 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2023-07-13T02:18:07.189 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/scratch_devs 2023-07-13T02:18:07.217 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi191:/dev/vg_nvme/lv_1... 2023-07-13T02:18:07.217 DEBUG:teuthology.orchestra.run.smithi191:> sudo nvme disconnect -n lv_1 2023-07-13T02:18:07.355 INFO:teuthology.orchestra.run.smithi191.stdout:NQN:lv_1 disconnected 1 controller(s) 2023-07-13T02:18:07.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-13T02:18:07.357 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi191:/dev/vg_nvme/lv_2... 2023-07-13T02:18:07.357 DEBUG:teuthology.orchestra.run.smithi191:> sudo nvme disconnect -n lv_2 2023-07-13T02:18:07.488 INFO:teuthology.orchestra.run.smithi191.stdout:NQN:lv_2 disconnected 1 controller(s) 2023-07-13T02:18:07.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-13T02:18:07.490 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi191:/dev/vg_nvme/lv_3... 2023-07-13T02:18:07.491 DEBUG:teuthology.orchestra.run.smithi191:> sudo nvme disconnect -n lv_3 2023-07-13T02:18:07.613 INFO:teuthology.orchestra.run.smithi191.stdout:NQN:lv_3 disconnected 1 controller(s) 2023-07-13T02:18:07.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-13T02:18:07.615 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi191:/dev/vg_nvme/lv_4... 2023-07-13T02:18:07.616 DEBUG:teuthology.orchestra.run.smithi191:> sudo nvme disconnect -n lv_4 2023-07-13T02:18:07.734 INFO:teuthology.orchestra.run.smithi191.stdout:NQN:lv_4 disconnected 1 controller(s) 2023-07-13T02:18:07.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-13T02:18:07.736 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-07-13T02:18:07.736 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/scratch_devs 2023-07-13T02:18:07.763 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-07-13T02:18:07.773 INFO:teuthology.task.clock:Checking final clock skew... 2023-07-13T02:18:07.773 DEBUG:teuthology.orchestra.run.smithi032:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-13T02:18:07.776 DEBUG:teuthology.orchestra.run.smithi191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-13T02:18:07.793 INFO:teuthology.orchestra.run.smithi032.stderr:bash: ntpq: command not found 2023-07-13T02:18:07.798 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-13T02:18:07.798 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2023-07-13T02:18:07.798 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 6 +1780us[+1780us] +/- 105ms 2023-07-13T02:18:07.799 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 8 -899us[ -899us] +/- 81ms 2023-07-13T02:18:07.799 INFO:teuthology.orchestra.run.smithi032.stdout:^* hv03.front.sepia.ceph.com 3 6 377 10 -386us[ -388us] +/- 74ms 2023-07-13T02:18:07.799 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-13T02:18:07.820 INFO:teuthology.orchestra.run.smithi191.stderr:bash: ntpq: command not found 2023-07-13T02:18:07.824 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-13T02:18:07.825 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2023-07-13T02:18:07.825 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 9 +1770us[+1770us] +/- 105ms 2023-07-13T02:18:07.825 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 11 -913us[ -913us] +/- 81ms 2023-07-13T02:18:07.825 INFO:teuthology.orchestra.run.smithi191.stdout:^* hv03.front.sepia.ceph.com 3 6 377 11 -388us[ -386us] +/- 74ms 2023-07-13T02:18:07.826 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-13T02:18:07.826 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-07-13T02:18:07.835 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-07-13T02:18:07.836 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-07-13T02:18:07.880 DEBUG:teuthology.orchestra.run.smithi032:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-13T02:18:07.982 DEBUG:teuthology.orchestra.run.smithi191:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-13T02:18:08.072 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-13T02:18:08.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-13T02:18:08.108 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-13T02:18:08.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-13T02:18:08.141 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-07-13T02:18:08.181 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-07-13T02:18:08.216 INFO:teuthology.task.internal:Duration was 1460.571951 seconds 2023-07-13T02:18:08.217 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-07-13T02:18:08.246 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-07-13T02:18:08.247 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-13T02:18:08.249 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-13T02:18:08.295 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-13T02:18:08.297 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-13T02:18:08.750 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-07-13T02:18:08.750 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi032.front.sepia.ceph.com 2023-07-13T02:18:08.750 DEBUG:teuthology.orchestra.run.smithi032:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-13T02:18:08.776 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi191.front.sepia.ceph.com 2023-07-13T02:18:08.777 DEBUG:teuthology.orchestra.run.smithi191:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-13T02:18:08.801 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-07-13T02:18:08.802 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-13T02:18:08.820 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-13T02:18:08.885 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-07-13T02:18:08.886 DEBUG:teuthology.orchestra.run.smithi032:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-13T02:18:08.888 DEBUG:teuthology.orchestra.run.smithi191:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-13T02:18:09.005 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-07-13T02:18:09.015 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-07-13T02:18:09.015 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-13T02:18:09.043 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-13T02:18:09.071 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-07-13T02:18:09.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-13T02:18:09.103 DEBUG:teuthology.orchestra.run.smithi191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-13T02:18:09.128 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = core 2023-07-13T02:18:09.138 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = core 2023-07-13T02:18:09.170 DEBUG:teuthology.orchestra.run.smithi032:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-13T02:18:09.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-13T02:18:09.215 DEBUG:teuthology.orchestra.run.smithi191:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-13T02:18:09.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-13T02:18:09.232 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-07-13T02:18:09.246 INFO:teuthology.task.internal:Transferring archived files... 2023-07-13T02:18:09.247 DEBUG:teuthology.misc:Transferring archived files from smithi032:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2023-07-13_00:38:31-orch:cephadm-cv-refactor-osd-objectstore-distro-default-smithi/7335617/remote/smithi032 2023-07-13T02:18:09.247 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-13T02:18:09.294 DEBUG:teuthology.misc:Transferring archived files from smithi191:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2023-07-13_00:38:31-orch:cephadm-cv-refactor-osd-objectstore-distro-default-smithi/7335617/remote/smithi191 2023-07-13T02:18:09.294 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-13T02:18:09.331 INFO:teuthology.task.internal:Removing archive directory... 2023-07-13T02:18:09.331 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-13T02:18:09.333 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-13T02:18:09.384 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-07-13T02:18:09.395 INFO:teuthology.task.internal:Not uploading archives. 2023-07-13T02:18:09.396 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-07-13T02:18:09.442 INFO:teuthology.task.internal:Tidying up after the test... 2023-07-13T02:18:09.442 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-13T02:18:09.445 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-13T02:18:09.461 INFO:teuthology.orchestra.run.smithi032.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 13 02:18 /home/ubuntu/cephtest 2023-07-13T02:18:09.462 INFO:teuthology.orchestra.run.smithi191.stdout: 525343 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 13 02:18 /home/ubuntu/cephtest 2023-07-13T02:18:09.463 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-07-13T02:18:09.496 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-07-13T02:18:09.655 INFO:teuthology.nuke:Checking targets against current locks 2023-07-13T02:18:09.686 DEBUG:teuthology.nuke:shortname: smithi032 2023-07-13T02:18:09.686 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-13T02:18:09.712 DEBUG:teuthology.nuke:shortname: smithi191 2023-07-13T02:18:09.712 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-13T02:18:09.715 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2023-07-13_00:38:31-orch:cephadm-cv-refactor-osd-objectstore-distro-default-smithi/7335617', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-13 01:42:57.447577', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkuioc2rB2/ZTZ+AFnxgD9c4H7/EH7hXBXfONd5YUN24Waof4JzkkRP1BqAM1FozKfo/ofJ+PzGw1gJmk9zmuqkLw3IOO/FiA0PL+GnZT778QaDf1dZghcPsnB5FfyWV1zutiz/anIzgnVhwMjUyjvPcdMxpM+7sj/1WH5w+GDWze5LUtrxdz8WsE8ksBqjzRojkDaekwrIiF5td3u6P0KAXXNJEDJA079oG4/RkIqvdQ528YZvsIaplCdOrxdW4cjLrw05CgRhaO1RsGF2CFj82/fokfTYoy411zJAvQi68IIWCSLSXXPKUHV+gPZzBsi1b8SyWVmntQdswCiGpcGTr+HgxRaYS/fXBGucFx3M9VOUB7aPUg9QF5ZdgBIRvdDCIiY3g3B9o8y/iLRNbfEqw1p3LYe3iSYXI0j3iUA4hH3Veaisaiv5BWwe1aP6dgv0X0e+JG5GyItClhDX5b/SXVZvlFX/s4QNpaoh1Kj+tkQ6Tk5Fc0+j2Hihm4hhWc='} 2023-07-13T02:18:09.740 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2023-07-13_00:38:31-orch:cephadm-cv-refactor-osd-objectstore-distro-default-smithi/7335617', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-13 01:42:57.446116', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHUDwEPbZnAEsrJfqXeQeXDFMxSMHER0Ti13d2UX1k+VHmo+1j2Adq//dylnODETskRCo01l8iJe1AZkohtm5uQOJNgLPgUdczL6qor26lqvvHVbffHAgqG8BpA732n7iRPqxjL1hmgAS/334HbwaDYHUdy+x59oI30zLCClOj+fYyIaEdX47LqCkO/TFZMIkZUeIVqPBSJjNnjgPtNp5I4qulMhsk9hjC4/rBnkvpFN8XMLdge4lqPwzc4aH4XEEjvLswlawEdUIVEZlnmH0Hzq7S9QliGBbQH2RFtHa2T+le+u6zhF/eZOe/TPwjBr5ZnvTSRRs7XlNAMRESBFJg9nkWcxHG4JPSyzP6tYgrxKhyq9TpMyh7LyzLLv2hBA/3Mljd85p5g9cI1GSZus1f2v/KY0hdt8HVghubPtdsQXpJod+DjBS38IUcePNKAK8MyRqayYJX2pu3zvXPW4SPhvW8bK9EDbRT3cDvg5ilnxAjegu6fppdXH9PpG/mwtc='} 2023-07-13T02:18:09.772 INFO:teuthology.orchestra.console:Power off smithi032 2023-07-13T02:18:09.773 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-13T02:18:09.796 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-13T02:18:09.821 INFO:teuthology.orchestra.console:Power off smithi191 2023-07-13T02:18:09.822 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-13T02:18:09.846 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-13T02:18:13.835 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-13T02:18:13.871 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-13T02:18:17.952 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-13T02:18:17.982 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-13T02:18:18.167 INFO:teuthology.orchestra.console:Power off for smithi032 completed 2023-07-13T02:18:22.096 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-13T02:18:22.307 INFO:teuthology.orchestra.console:Power off for smithi191 completed 2023-07-13T02:18:22.412 INFO:teuthology.run:Summary data: description: orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} duration: 1460.5719511508942 failure_reason: reached maximum tries (121) after waiting for 120 seconds owner: scheduled_gabrioux@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=7f82ea8e3a5d4ee49421563f21eb24a7 status: fail success: false 2023-07-13T02:18:22.413 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-13T02:18:22.499 INFO:teuthology.run:FAIL