2023-10-09T07:27:37.555 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-09T07:27:37.556 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_bdavidov@teuthology --description rbd:nvmeof/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_nvmeof} --archive /home/teuthworker/archive/bdavidov-2023-10-09_07:20:03-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7418391 --verbose --name bdavidov-2023-10-09_07:20:03-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi -- /home/teuthworker/archive/bdavidov-2023-10-09_07:20:03-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7418391/orig.config.yaml 2023-10-09T07:27:37.593 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-09T07:27:37.685 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/bdavidov-2023-10-09_07:20:03-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7418391 branch: wip-barakda-cephadm-nvmeof description: rbd:nvmeof/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_nvmeof} email: null first_in_suite: false job_id: '7418391' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: bdavidov-2023-10-09_07:20:03-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 30 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-barakda-cephadm-nvmeof ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: bcea8c48517ff154f392584022efa0718f51c9f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: bcea8c48517ff154f392584022efa0718f51c9f2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-nvmetest-teuthology sha1: 6baafb588e77fa3b88e89ecedcf1f86c473eacbf owner: scheduled_bdavidov@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - mon.a - mgr.x - osd.0 - osd.1 - client.0 seed: 9138 sha1: bcea8c48517ff154f392584022efa0718f51c9f2 sleep_before_teardown: 0 suite: rbd:nvmeof suite_branch: wip-nvmetest-teuthology suite_path: /home/teuthworker/src/github.com_barakda_ceph_6baafb588e77fa3b88e89ecedcf1f86c473eacbf/qa suite_relpath: qa suite_repo: https://github.com/barakda/ceph suite_sha1: 6baafb588e77fa3b88e89ecedcf1f86c473eacbf targets: smithi115.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLzLt28sfKk++bxLIyz6AKjhC+jrWdfJvnbtygRh8hEIH11qIZgeVyMsC/Q+TKMwk01YkQ7vOdbRI0aiDX/tsNA= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - sudo sed -i 's/runtime = "runc"/#runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/#runtime = "crun"/runtime = "crun"/g' /usr/share/containers/containers.conf - cephadm: null - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph osd lspools - ceph osd pool create mypool - ceph osd pool application enable mypool rbd - ceph osd lspools - ceph osd pool ls detail - sleep 60 - ceph config set mgr mgr/cephadm/container_image_nvmeof quay.io/ceph/nvmeof:0.0.4 - sleep 60 - ceph config get mgr mgr/cephadm/container_image_nvmeof - ceph orch apply nvmeof mypool --placement="1 $(hostname)" - ceph orch ps --refresh - ceph versions - ceph -s - ceph orch ls - rbd create mypool/myimage --size 8Gi - rbd ls mypool - 'set -ex # cat /proc/cpuinfo ' - exec: host.a: - 'set -ex # # print nvmeof start up logs s=$(systemctl list-units | grep nvmeof.mypool | awk ''{print $1}'') echo "<---- s variable is [ $s ]---->" PAGER=cat journalctl -u $s ' - cephadm.wait_for_service: service: nvmeof.mypool - exec: host.a: - modprobe nvme-fabrics - 'set -ex # # print nvmeof start up logs s=$(systemctl list-units | grep nvmeof.mypool | awk ''{print $1}'') echo "<---- s variable is [ $s ]---->" PAGER=cat journalctl -u $s GATEWAY=$(echo $s | cut -d "@" -f2- | sed ''s/.\{8\}$//'') echo "<---- GATEWAY is [ $GATEWAY ]---->" IP=$(echo $(hostname -I) | cut -d '' '' -f1) HOSTNAME=$(hostname) IMAGE="quay.io/ceph/nvmeof-cli:0.0.4" POOL="mypool" MIMAGE="myimage" BDEV="mybdev" SERIAL="SPDK00000000000001" NQN="nqn.2016-06.io.spdk:cnode1" PORT="4420" SRPORT="5500" podman run -it $IMAGE --server-address $IP --server-port $SRPORT create_bdev --pool $POOL --image $MIMAGE --bdev $BDEV podman images podman ps podman run -it $IMAGE --server-address $IP --server-port $SRPORT create_subsystem --subnqn $NQN --serial $SERIAL podman run -it $IMAGE --server-address $IP --server-port $SRPORT add_namespace --subnqn $NQN --bdev $BDEV podman run -it $IMAGE --server-address $IP --server-port $SRPORT create_listener -n $NQN -g $GATEWAY -a $IP -s $PORT podman run -it $IMAGE --server-address $IP --server-port $SRPORT add_host --subnqn $NQN --host "*" podman run -it $IMAGE --server-address $IP --server-port $SRPORT get_subsystems nvme discover -t tcp -a $IP -s $PORT nvme connect -t tcp --traddr $IP -s $PORT -n $NQN nvme list ' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph osd lspools - 'set -ex s=$(ceph orch ps | grep nvmeof.mypool | awk ''{print $4}'') echo $s [ $s = running ] ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-10-09_07:20:03 tube: smithi use_shaman: true user: bdavidov verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-09T07:27:37.685 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_barakda_ceph_6baafb588e77fa3b88e89ecedcf1f86c473eacbf/qa; will attempt to use it 2023-10-09T07:27:37.686 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_barakda_ceph_6baafb588e77fa3b88e89ecedcf1f86c473eacbf/qa/tasks 2023-10-09T07:27:37.686 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-09T07:27:37.688 INFO:teuthology.task.internal:Checking packages... 2023-10-09T07:27:37.710 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bcea8c48517ff154f392584022efa0718f51c9f2' 2023-10-09T07:27:37.710 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-09T07:27:37.710 INFO:teuthology.packaging:ref: None 2023-10-09T07:27:37.711 INFO:teuthology.packaging:tag: None 2023-10-09T07:27:37.711 INFO:teuthology.packaging:branch: wip-barakda-cephadm-nvmeof 2023-10-09T07:27:37.711 INFO:teuthology.packaging:sha1: bcea8c48517ff154f392584022efa0718f51c9f2 2023-10-09T07:27:37.711 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-barakda-cephadm-nvmeof 2023-10-09T07:27:37.855 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-5817.gbcea8c48 2023-10-09T07:27:37.855 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-09T07:27:37.862 INFO:teuthology.task.internal:no buildpackages task found 2023-10-09T07:27:37.862 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-09T07:27:37.869 INFO:teuthology.task.internal:Saving configuration 2023-10-09T07:27:37.882 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-09T07:27:37.888 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-09T07:27:37.913 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/bdavidov-2023-10-09_07:20:03-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7418391', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-09 07:22:16.923236', 'locked_by': 'scheduled_bdavidov@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLzLt28sfKk++bxLIyz6AKjhC+jrWdfJvnbtygRh8hEIH11qIZgeVyMsC/Q+TKMwk01YkQ7vOdbRI0aiDX/tsNA='} 2023-10-09T07:27:37.914 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-09T07:27:37.920 INFO:teuthology.task.internal:roles: ubuntu@smithi115.front.sepia.ceph.com - ['host.a', 'mon.a', 'mgr.x', 'osd.0', 'osd.1', 'client.0'] 2023-10-09T07:27:37.920 INFO:teuthology.run_tasks:Running task console_log... 2023-10-09T07:27:37.960 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5c91aa0b80>, signals=[15]) 2023-10-09T07:27:37.961 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-09T07:27:37.967 INFO:teuthology.task.internal:Opening connections... 2023-10-09T07:27:37.967 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi115.front.sepia.ceph.com 2023-10-09T07:27:37.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-09T07:27:38.036 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-09T07:27:38.042 DEBUG:teuthology.orchestra.run.smithi115:> uname -m 2023-10-09T07:27:38.111 INFO:teuthology.orchestra.run.smithi115.stdout:x86_64 2023-10-09T07:27:38.111 DEBUG:teuthology.orchestra.run.smithi115:> cat /etc/os-release 2023-10-09T07:27:38.171 INFO:teuthology.orchestra.run.smithi115.stdout:NAME="CentOS Stream" 2023-10-09T07:27:38.172 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION="8" 2023-10-09T07:27:38.172 INFO:teuthology.orchestra.run.smithi115.stdout:ID="centos" 2023-10-09T07:27:38.172 INFO:teuthology.orchestra.run.smithi115.stdout:ID_LIKE="rhel fedora" 2023-10-09T07:27:38.172 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_ID="8" 2023-10-09T07:27:38.172 INFO:teuthology.orchestra.run.smithi115.stdout:PLATFORM_ID="platform:el8" 2023-10-09T07:27:38.172 INFO:teuthology.orchestra.run.smithi115.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-09T07:27:38.172 INFO:teuthology.orchestra.run.smithi115.stdout:ANSI_COLOR="0;31" 2023-10-09T07:27:38.172 INFO:teuthology.orchestra.run.smithi115.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-09T07:27:38.172 INFO:teuthology.orchestra.run.smithi115.stdout:HOME_URL="https://centos.org/" 2023-10-09T07:27:38.172 INFO:teuthology.orchestra.run.smithi115.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-09T07:27:38.172 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-09T07:27:38.172 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-09T07:27:38.173 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server 2023-10-09T07:27:38.195 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-09T07:27:38.204 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-09T07:27:38.210 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-09T07:27:38.211 DEBUG:teuthology.orchestra.run.smithi115:> test '!' -e /home/ubuntu/cephtest 2023-10-09T07:27:38.231 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-09T07:27:38.237 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-09T07:27:38.237 DEBUG:teuthology.orchestra.run.smithi115:> test -z $(ls -A /var/lib/ceph) 2023-10-09T07:27:38.300 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-09T07:27:38.333 INFO:teuthology.run_tasks:Running task kernel... 2023-10-09T07:27:38.351 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-09T07:27:38.351 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-09T07:27:38.351 DEBUG:teuthology.orchestra.run.smithi115:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-09T07:27:38.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-09T07:27:38.370 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2023-10-09T07:27:38.430 INFO:teuthology.orchestra.run.smithi115.stdout:4.18.0-489.el8.x86_64 2023-10-09T07:27:38.430 INFO:teuthology.task.kernel:Running kernel on smithi115: 4.18.0-489.el8.x86_64 2023-10-09T07:27:38.430 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum install -y kernel 2023-10-09T07:27:45.790 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - AppStream 7.1 MB/s | 33 MB 00:04 2023-10-09T07:28:01.758 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - BaseOS 4.8 MB/s | 48 MB 00:09 2023-10-09T07:28:09.570 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - Extras 19 kB/s | 18 kB 00:00 2023-10-09T07:28:10.087 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - Extras common packages 21 kB/s | 6.9 kB 00:00 2023-10-09T07:28:10.397 INFO:teuthology.orchestra.run.smithi115.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-10-09T07:28:11.483 INFO:teuthology.orchestra.run.smithi115.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2023-10-09T07:28:15.316 INFO:teuthology.orchestra.run.smithi115.stdout:lab-extras 442 kB/s | 24 kB 00:00 2023-10-09T07:28:18.878 INFO:teuthology.orchestra.run.smithi115.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-10-09T07:28:18.878 INFO:teuthology.orchestra.run.smithi115.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-10-09T07:28:19.115 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-10-09T07:28:19.115 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-10-09T07:28:19.115 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2023-10-09T07:28:19.115 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-10-09T07:28:19.115 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-10-09T07:28:19.116 INFO:teuthology.orchestra.run.smithi115.stdout: kernel x86_64 4.18.0-514.el8 baseos 10 M 2023-10-09T07:28:19.116 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2023-10-09T07:28:19.116 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-core x86_64 4.18.0-514.el8 baseos 43 M 2023-10-09T07:28:19.116 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules x86_64 4.18.0-514.el8 baseos 35 M 2023-10-09T07:28:19.116 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:28:19.116 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-10-09T07:28:19.116 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-10-09T07:28:19.116 INFO:teuthology.orchestra.run.smithi115.stdout:Install 3 Packages 2023-10-09T07:28:19.116 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:28:19.117 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 88 M 2023-10-09T07:28:19.117 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 96 M 2023-10-09T07:28:19.117 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-10-09T07:28:20.650 INFO:teuthology.orchestra.run.smithi115.stdout:(1/3): kernel-4.18.0-514.el8.x86_64.rpm 6.7 MB/s | 10 MB 00:01 2023-10-09T07:28:22.317 INFO:teuthology.orchestra.run.smithi115.stdout:(2/3): kernel-modules-4.18.0-514.el8.x86_64.rpm 11 MB/s | 35 MB 00:03 2023-10-09T07:28:22.867 INFO:teuthology.orchestra.run.smithi115.stdout:(3/3): kernel-core-4.18.0-514.el8.x86_64.rpm 11 MB/s | 43 MB 00:03 2023-10-09T07:28:22.867 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-10-09T07:28:22.867 INFO:teuthology.orchestra.run.smithi115.stdout:Total 23 MB/s | 88 MB 00:03 2023-10-09T07:28:23.534 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-10-09T07:28:23.954 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-10-09T07:28:23.955 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-10-09T07:28:25.175 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-10-09T07:28:25.194 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-10-09T07:28:27.920 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-10-09T07:28:28.438 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-core-4.18.0-514.el8.x86_64 1/3 2023-10-09T07:28:31.977 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 1/3 2023-10-09T07:28:32.403 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-10-09T07:28:35.776 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-10-09T07:28:36.133 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-4.18.0-514.el8.x86_64 3/3 2023-10-09T07:28:57.970 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 3/3 2023-10-09T07:28:57.994 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-10-09T07:28:58.386 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-4.18.0-514.el8.x86_64 3/3 2023-10-09T07:28:58.387 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-4.18.0-514.el8.x86_64 1/3 2023-10-09T07:28:58.387 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-core-4.18.0-514.el8.x86_64 2/3 2023-10-09T07:28:59.113 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-10-09T07:28:59.114 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:28:59.114 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-10-09T07:28:59.114 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-4.18.0-514.el8.x86_64 kernel-core-4.18.0-514.el8.x86_64 2023-10-09T07:28:59.114 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules-4.18.0-514.el8.x86_64 2023-10-09T07:28:59.114 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:28:59.114 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-10-09T07:28:59.263 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q kernel | sort -rV | head -n 1 2023-10-09T07:28:59.478 INFO:teuthology.orchestra.run.smithi115.stdout:kernel-4.18.0-514.el8.x86_64 2023-10-09T07:28:59.478 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-10-09T07:28:59.478 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-514.el8.x86_64 2023-10-09T07:28:59.478 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-09T07:28:59.479 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-09T07:28:59.479 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-10-09T07:28:59.479 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi115.front.sepia.ceph.com, path=None, version=distro) 2023-10-09T07:28:59.479 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q kernel | sort -rV | head -n 1 2023-10-09T07:28:59.685 INFO:teuthology.orchestra.run.smithi115.stdout:kernel-4.18.0-514.el8.x86_64 2023-10-09T07:28:59.686 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-10-09T07:28:59.686 DEBUG:teuthology.orchestra.run.smithi115:> sudo rpm -qi grub2-tools 2023-10-09T07:28:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:Name : grub2-tools 2023-10-09T07:28:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:Epoch : 1 2023-10-09T07:28:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:Version : 2.02 2023-10-09T07:28:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:Release : 129.el8 2023-10-09T07:28:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:Architecture: x86_64 2023-10-09T07:28:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-09T07:28:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:Group : System Environment/Base 2023-10-09T07:28:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:Size : 9265586 2023-10-09T07:28:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:License : GPLv3+ 2023-10-09T07:28:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-09T07:28:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-09T07:28:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-09T07:28:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-09T07:28:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:Relocations : (not relocatable) 2023-10-09T07:28:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:Packager : CentOS BuildSystem 2023-10-09T07:28:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:Vendor : CentOS 2023-10-09T07:28:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-09T07:28:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:Summary : Support tools for GRUB. 2023-10-09T07:28:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:Description : 2023-10-09T07:28:59.720 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:28:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-09T07:28:59.721 INFO:teuthology.orchestra.run.smithi115.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-09T07:28:59.721 INFO:teuthology.orchestra.run.smithi115.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-09T07:28:59.721 INFO:teuthology.orchestra.run.smithi115.stdout:hardware devices. 2023-10-09T07:28:59.721 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:28:59.721 INFO:teuthology.orchestra.run.smithi115.stdout:This subpackage provides tools for support of all platforms. 2023-10-09T07:28:59.722 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-09T07:28:59.722 INFO:teuthology.task.kernel:Updating grub on smithi115 to boot 4.18.0-514.el8.x86_64 2023-10-09T07:28:59.722 DEBUG:teuthology.orchestra.run.smithi115:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-09T07:29:01.165 INFO:teuthology.orchestra.run.smithi115.stderr:Generating grub configuration file ... 2023-10-09T07:29:03.439 INFO:teuthology.orchestra.run.smithi115.stderr:done 2023-10-09T07:29:03.441 DEBUG:teuthology.orchestra.run.smithi115:> mktemp 2023-10-09T07:29:03.474 INFO:teuthology.orchestra.run.smithi115.stdout:/tmp/tmp.zVsd5GnIkN 2023-10-09T07:29:03.475 DEBUG:teuthology.orchestra.run.smithi115:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zVsd5GnIkN 2023-10-09T07:29:03.565 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0666 /tmp/tmp.zVsd5GnIkN 2023-10-09T07:29:03.664 DEBUG:teuthology.orchestra.remote:smithi115:/tmp/tmp.zVsd5GnIkN is 6KB 2023-10-09T07:29:03.713 DEBUG:teuthology.orchestra.run.smithi115:> rm -fr /tmp/tmp.zVsd5GnIkN 2023-10-09T07:29:03.730 DEBUG:teuthology.orchestra.run.smithi115:> sudo /bin/ls /boot/loader/entries || true 2023-10-09T07:29:03.811 INFO:teuthology.orchestra.run.smithi115.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-09T07:29:03.811 INFO:teuthology.orchestra.run.smithi115.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-09T07:29:03.811 INFO:teuthology.orchestra.run.smithi115.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-09T07:29:03.811 INFO:teuthology.orchestra.run.smithi115.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64.conf 2023-10-09T07:29:03.812 DEBUG:teuthology.orchestra.run.smithi115:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64 2023-10-09T07:29:03.951 DEBUG:teuthology.orchestra.run.smithi115:> sudo shutdown -r now 2023-10-09T07:29:33.982 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-10-09T07:29:33.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2023-10-09T07:29:33.983 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-09T07:30:08.638 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2023-10-09T07:30:17.646 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2023-10-09T07:30:17.647 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-09T07:30:36.026 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2023-10-09T07:30:48.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2023-10-09T07:30:48.039 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-09T07:30:48.624 DEBUG:teuthology.orchestra.run.smithi115:> true 2023-10-09T07:30:48.980 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2023-10-09T07:30:48.980 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-514.el8.x86_64"... 2023-10-09T07:30:48.981 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2023-10-09T07:30:49.000 INFO:teuthology.orchestra.run.smithi115.stdout:4.18.0-514.el8.x86_64 2023-10-09T07:30:49.000 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-514.el8.x86_64 vs 4.18.0-514.el8.x86_64 2023-10-09T07:30:49.000 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-09T07:30:49.000 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-09T07:30:50.001 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-10-09T07:30:50.001 DEBUG:teuthology.orchestra.run.smithi115:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-09T07:30:50.121 INFO:teuthology.orchestra.run.smithi115.stdout:ttyS1 2023-10-09T07:30:50.137 DEBUG:teuthology.parallel:result is None 2023-10-09T07:30:50.137 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-09T07:30:50.144 INFO:teuthology.task.internal:Creating test directory... 2023-10-09T07:30:50.144 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-09T07:30:50.162 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-09T07:30:50.309 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-09T07:30:50.323 INFO:teuthology.task.internal:Creating archive directory... 2023-10-09T07:30:50.324 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-09T07:30:50.350 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-09T07:30:50.359 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-09T07:30:50.359 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-09T07:30:50.423 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-09T07:30:50.434 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-09T07:30:50.435 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-09T07:30:50.443 INFO:teuthology.task.internal:Configuring sudo... 2023-10-09T07:30:50.443 DEBUG:teuthology.orchestra.run.smithi115:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-09T07:30:50.505 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-09T07:30:50.515 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-09T07:30:50.515 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-09T07:30:50.563 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-09T07:30:50.646 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-09T07:30:50.698 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-09T07:30:50.781 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-09T07:30:50.825 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-10-09T07:30:50.825 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-09T07:30:50.901 DEBUG:teuthology.orchestra.run.smithi115:> sudo service rsyslog restart 2023-10-09T07:30:51.006 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-09T07:30:51.324 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-09T07:30:51.332 INFO:teuthology.task.internal:Starting timer... 2023-10-09T07:30:51.332 INFO:teuthology.run_tasks:Running task pcp... 2023-10-09T07:30:51.346 INFO:teuthology.run_tasks:Running task selinux... 2023-10-09T07:30:51.356 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-09T07:30:51.356 DEBUG:teuthology.orchestra.run.smithi115:> sudo service auditd rotate 2023-10-09T07:30:51.444 INFO:teuthology.orchestra.run.smithi115.stdout:Rotating logs: 2023-10-09T07:30:51.446 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-09T07:30:51.446 DEBUG:teuthology.orchestra.run.smithi115:> /usr/sbin/getenforce 2023-10-09T07:30:51.472 INFO:teuthology.orchestra.run.smithi115.stdout:Permissive 2023-10-09T07:30:51.473 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi115.front.sepia.ceph.com': 'permissive'} 2023-10-09T07:30:51.473 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-09T07:30:51.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-09T07:30:51.540 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-09T07:30:51.540 DEBUG:teuthology.orchestra.run.smithi115:> sudo /usr/sbin/setenforce permissive 2023-10-09T07:30:51.608 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-09T07:30:51.617 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-09T07:30:51.617 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-09T07:30:51.634 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-09T07:30:51.635 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi115.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-09T07:38:10.913 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi115.front.sepia.ceph.com')] 2023-10-09T07:38:10.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2023-10-09T07:38:10.915 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-09T07:38:10.987 DEBUG:teuthology.orchestra.run.smithi115:> true 2023-10-09T07:38:11.012 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2023-10-09T07:38:11.013 INFO:teuthology.run_tasks:Running task clock... 2023-10-09T07:38:11.022 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-09T07:38:11.022 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-09T07:38:11.023 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-09T07:38:11.085 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-09T07:38:11.098 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-09T07:38:11.161 INFO:teuthology.orchestra.run.smithi115.stderr:sudo: ntpd: command not found 2023-10-09T07:38:11.173 INFO:teuthology.orchestra.run.smithi115.stdout:506 Cannot talk to daemon 2023-10-09T07:38:11.187 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-09T07:38:11.201 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-09T07:38:11.265 INFO:teuthology.orchestra.run.smithi115.stderr:bash: ntpq: command not found 2023-10-09T07:38:11.269 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-09T07:38:11.269 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2023-10-09T07:38:11.269 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-09T07:38:11.269 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-09T07:38:11.269 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-09T07:38:11.269 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-09T07:38:11.270 INFO:teuthology.run_tasks:Running task pexec... 2023-10-09T07:38:11.279 INFO:teuthology.task.pexec:Executing custom commands... 2023-10-09T07:38:11.279 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi115.front.sepia.ceph.com 2023-10-09T07:38:11.280 DEBUG:teuthology.orchestra.run.smithi115:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-10-09T07:38:12.185 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:34 ago on Mon 09 Oct 2023 07:36:38 AM UTC. 2023-10-09T07:38:14.213 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-10-09T07:38:14.215 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-10-09T07:38:14.215 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2023-10-09T07:38:14.215 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-10-09T07:38:14.215 INFO:teuthology.orchestra.run.smithi115.stdout:Resetting modules: 2023-10-09T07:38:14.215 INFO:teuthology.orchestra.run.smithi115.stdout: container-tools 2023-10-09T07:38:14.215 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:38:14.215 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-10-09T07:38:14.215 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-10-09T07:38:14.215 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:38:14.468 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-10-09T07:38:15.401 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:37 ago on Mon 09 Oct 2023 07:36:38 AM UTC. 2023-10-09T07:38:17.412 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-10-09T07:38:17.566 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-10-09T07:38:17.566 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-10-09T07:38:17.567 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-10-09T07:38:17.567 INFO:teuthology.orchestra.run.smithi115.stdout:Installing group/module packages: 2023-10-09T07:38:17.567 INFO:teuthology.orchestra.run.smithi115.stdout: aardvark-dns x86_64 2:1.7.0-1.module_el8+487+8e42a277 appstream 1.0 M 2023-10-09T07:38:17.567 INFO:teuthology.orchestra.run.smithi115.stdout: buildah x86_64 1:1.31.3-1.module_el8+664+4072b3ae appstream 8.8 M 2023-10-09T07:38:17.567 INFO:teuthology.orchestra.run.smithi115.stdout: cockpit-podman noarch 75-1.module_el8+664+4072b3ae appstream 739 k 2023-10-09T07:38:17.567 INFO:teuthology.orchestra.run.smithi115.stdout: crun x86_64 1.8.7-1.module_el8+661+d1afb926 appstream 239 k 2023-10-09T07:38:17.567 INFO:teuthology.orchestra.run.smithi115.stdout: netavark x86_64 2:1.7.0-1.module_el8+487+8e42a277 appstream 3.7 M 2023-10-09T07:38:17.567 INFO:teuthology.orchestra.run.smithi115.stdout: python3-podman noarch 4.6.0-1.module_el8+582+a2558ae2 appstream 177 k 2023-10-09T07:38:17.567 INFO:teuthology.orchestra.run.smithi115.stdout: skopeo x86_64 2:1.13.3-1.module_el8+664+4072b3ae appstream 8.1 M 2023-10-09T07:38:17.567 INFO:teuthology.orchestra.run.smithi115.stdout: toolbox x86_64 0.0.99.4-5.module_el8+649+e2deac1d appstream 2.5 M 2023-10-09T07:38:17.567 INFO:teuthology.orchestra.run.smithi115.stdout: udica noarch 0.2.6-20.module_el8+487+8e42a277 appstream 53 k 2023-10-09T07:38:17.567 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2023-10-09T07:38:17.568 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-10-09T07:38:17.568 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-10-09T07:38:17.568 INFO:teuthology.orchestra.run.smithi115.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-10-09T07:38:17.568 INFO:teuthology.orchestra.run.smithi115.stdout:Installing weak dependencies: 2023-10-09T07:38:17.568 INFO:teuthology.orchestra.run.smithi115.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-10-09T07:38:17.568 INFO:teuthology.orchestra.run.smithi115.stdout:Installing module profiles: 2023-10-09T07:38:17.568 INFO:teuthology.orchestra.run.smithi115.stdout: container-tools/common 2023-10-09T07:38:17.568 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:38:17.568 INFO:teuthology.orchestra.run.smithi115.stdout:Enabling module streams: 2023-10-09T07:38:17.568 INFO:teuthology.orchestra.run.smithi115.stdout: container-tools rhel8 2023-10-09T07:38:17.568 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:38:17.568 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-10-09T07:38:17.568 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-10-09T07:38:17.569 INFO:teuthology.orchestra.run.smithi115.stdout:Install 13 Packages 2023-10-09T07:38:17.569 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:38:17.569 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 26 M 2023-10-09T07:38:17.569 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 87 M 2023-10-09T07:38:17.569 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-10-09T07:38:17.971 INFO:teuthology.orchestra.run.smithi115.stdout:(1/13): cockpit-podman-75-1.module_el8+664+4072 3.3 MB/s | 739 kB 00:00 2023-10-09T07:38:18.038 INFO:teuthology.orchestra.run.smithi115.stdout:(2/13): aardvark-dns-1.7.0-1.module_el8+487+8e4 3.5 MB/s | 1.0 MB 00:00 2023-10-09T07:38:18.064 INFO:teuthology.orchestra.run.smithi115.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 410 kB/s | 38 kB 00:00 2023-10-09T07:38:18.097 INFO:teuthology.orchestra.run.smithi115.stdout:(4/13): crun-1.8.7-1.module_el8+661+d1afb926.x8 4.0 MB/s | 239 kB 00:00 2023-10-09T07:38:18.156 INFO:teuthology.orchestra.run.smithi115.stdout:(5/13): python3-podman-4.6.0-1.module_el8+582+a 3.0 MB/s | 177 kB 00:00 2023-10-09T07:38:18.206 INFO:teuthology.orchestra.run.smithi115.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 494 kB/s | 25 kB 00:00 2023-10-09T07:38:18.265 INFO:teuthology.orchestra.run.smithi115.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.6 MB/s | 94 kB 00:00 2023-10-09T07:38:18.449 INFO:teuthology.orchestra.run.smithi115.stdout:(8/13): netavark-1.7.0-1.module_el8+487+8e42a27 9.7 MB/s | 3.7 MB 00:00 2023-10-09T07:38:18.690 INFO:teuthology.orchestra.run.smithi115.stdout:(9/13): toolbox-0.0.99.4-5.module_el8+649+e2dea 10 MB/s | 2.5 MB 00:00 2023-10-09T07:38:18.791 INFO:teuthology.orchestra.run.smithi115.stdout:(10/13): buildah-1.31.3-1.module_el8+664+4072b3 8.5 MB/s | 8.8 MB 00:01 2023-10-09T07:38:18.816 INFO:teuthology.orchestra.run.smithi115.stdout:(11/13): udica-0.2.6-20.module_el8+487+8e42a277 420 kB/s | 53 kB 00:00 2023-10-09T07:38:18.841 INFO:teuthology.orchestra.run.smithi115.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 816 kB/s | 41 kB 00:00 2023-10-09T07:38:19.092 INFO:teuthology.orchestra.run.smithi115.stdout:(13/13): skopeo-1.13.3-1.module_el8+664+4072b3a 9.9 MB/s | 8.1 MB 00:00 2023-10-09T07:38:19.092 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-10-09T07:38:19.092 INFO:teuthology.orchestra.run.smithi115.stdout:Total 17 MB/s | 26 MB 00:01 2023-10-09T07:38:19.355 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-10-09T07:38:19.375 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-10-09T07:38:19.376 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-10-09T07:38:19.616 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-10-09T07:38:19.619 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-10-09T07:38:20.307 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-10-09T07:38:20.561 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-10-09T07:38:20.731 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-10-09T07:38:20.945 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-10-09T07:38:21.210 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-10-09T07:38:21.784 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86 5/13 2023-10-09T07:38:22.691 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 6/13 2023-10-09T07:38:22.929 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 7/13 2023-10-09T07:38:23.142 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : crun-1.8.7-1.module_el8+661+d1afb926.x86_64 8/13 2023-10-09T07:38:23.429 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-podman-4.6.0-1.module_el8+582+a2558ae2.noa 9/13 2023-10-09T07:38:23.843 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : udica-0.2.6-20.module_el8+487+8e42a277.noarch 10/13 2023-10-09T07:38:24.742 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 11/13 2023-10-09T07:38:24.972 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 12/13 2023-10-09T07:38:25.197 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 13/13 2023-10-09T07:38:25.628 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 13/13 2023-10-09T07:38:25.628 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86 1/13 2023-10-09T07:38:25.628 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 2/13 2023-10-09T07:38:25.629 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 3/13 2023-10-09T07:38:25.629 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-10-09T07:38:25.629 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : crun-1.8.7-1.module_el8+661+d1afb926.x86_64 5/13 2023-10-09T07:38:25.629 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 6/13 2023-10-09T07:38:25.629 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-podman-4.6.0-1.module_el8+582+a2558ae2.noa 7/13 2023-10-09T07:38:25.629 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-10-09T07:38:25.629 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-10-09T07:38:25.629 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 10/13 2023-10-09T07:38:25.629 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 11/13 2023-10-09T07:38:25.629 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : udica-0.2.6-20.module_el8+487+8e42a277.noarch 12/13 2023-10-09T07:38:26.427 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-10-09T07:38:26.428 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:38:26.428 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-10-09T07:38:26.428 INFO:teuthology.orchestra.run.smithi115.stdout: aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86_64 2023-10-09T07:38:26.428 INFO:teuthology.orchestra.run.smithi115.stdout: buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 2023-10-09T07:38:26.428 INFO:teuthology.orchestra.run.smithi115.stdout: cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 2023-10-09T07:38:26.429 INFO:teuthology.orchestra.run.smithi115.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-10-09T07:38:26.429 INFO:teuthology.orchestra.run.smithi115.stdout: crun-1.8.7-1.module_el8+661+d1afb926.x86_64 2023-10-09T07:38:26.429 INFO:teuthology.orchestra.run.smithi115.stdout: netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 2023-10-09T07:38:26.429 INFO:teuthology.orchestra.run.smithi115.stdout: python3-podman-4.6.0-1.module_el8+582+a2558ae2.noarch 2023-10-09T07:38:26.429 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-10-09T07:38:26.429 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-10-09T07:38:26.430 INFO:teuthology.orchestra.run.smithi115.stdout: skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 2023-10-09T07:38:26.430 INFO:teuthology.orchestra.run.smithi115.stdout: toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 2023-10-09T07:38:26.430 INFO:teuthology.orchestra.run.smithi115.stdout: udica-0.2.6-20.module_el8+487+8e42a277.noarch 2023-10-09T07:38:26.430 INFO:teuthology.orchestra.run.smithi115.stdout: yajl-2.1.0-12.el8.x86_64 2023-10-09T07:38:26.430 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:38:26.430 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-10-09T07:38:26.717 DEBUG:teuthology.parallel:result is None 2023-10-09T07:38:26.718 INFO:teuthology.run_tasks:Running task cephadm... 2023-10-09T07:38:26.790 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'bcea8c48517ff154f392584022efa0718f51c9f2'} 2023-10-09T07:38:26.791 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 2023-10-09T07:38:26.791 INFO:tasks.cephadm:Cluster fsid is d56ccb5c-6676-11ee-8db5-212e2dc638e7 2023-10-09T07:38:26.791 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-10-09T07:38:26.791 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.115'} 2023-10-09T07:38:26.792 INFO:tasks.cephadm:First mon is mon.a on smithi115 2023-10-09T07:38:26.792 INFO:tasks.cephadm:First mgr is x 2023-10-09T07:38:26.792 INFO:tasks.cephadm:Normalizing hostnames... 2023-10-09T07:38:26.792 DEBUG:teuthology.orchestra.run.smithi115:> sudo hostname $(hostname -s) 2023-10-09T07:38:26.821 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-10-09T07:38:26.821 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=bcea8c48517ff154f392584022efa0718f51c9f2 2023-10-09T07:38:27.059 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'bcea8c48517ff154f392584022efa0718f51c9f2', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/72782/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-5817-gbcea8c48', 'node_name': '172.21.4.65+confusa03', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-5817.gbcea8c48'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-barakda-cephadm-nvmeof/bcea8c48517ff154f392584022efa0718f51c9f2/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-09-25 21:06:38.915379', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-barakda-cephadm-nvmeof', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-barakda-cephadm-nvmeof/bcea8c48517ff154f392584022efa0718f51c9f2/centos/8/flavors/default/', 'archs': ['x86_64', 'source', 'arm64'], 'distro': 'centos'}] 2023-10-09T07:38:27.275 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-barakda-cephadm-nvmeof, sha1 bcea8c48517ff154f392584022efa0718f51c9f2 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=bcea8c48517ff154f392584022efa0718f51c9f2 2023-10-09T07:38:27.275 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-barakda-cephadm-nvmeof/bcea8c48517ff154f392584022efa0718f51c9f2/centos/8/x86_64/flavors/default/cephadm 2023-10-09T07:38:27.276 DEBUG:teuthology.orchestra.run.smithi115:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-barakda-cephadm-nvmeof/bcea8c48517ff154f392584022efa0718f51c9f2/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-10-09T07:38:28.238 INFO:teuthology.orchestra.run.smithi115.stdout:-rw-rw-r--. 1 ubuntu ubuntu 705688 Oct 9 07:38 /home/ubuntu/cephtest/cephadm 2023-10-09T07:38:28.238 DEBUG:teuthology.orchestra.run.smithi115:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-10-09T07:38:28.267 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 on all hosts... 2023-10-09T07:38:28.267 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 pull 2023-10-09T07:38:30.204 INFO:teuthology.orchestra.run.smithi115.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2... 2023-10-09T07:38:58.283 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-10-09T07:38:58.283 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_version": "ceph version 18.0.0-5817-gbcea8c48 (bcea8c48517ff154f392584022efa0718f51c9f2) reef (dev)", 2023-10-09T07:38:58.283 INFO:teuthology.orchestra.run.smithi115.stdout: "image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", 2023-10-09T07:38:58.283 INFO:teuthology.orchestra.run.smithi115.stdout: "repo_digests": [ 2023-10-09T07:38:58.283 INFO:teuthology.orchestra.run.smithi115.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7" 2023-10-09T07:38:58.283 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2023-10-09T07:38:58.283 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-10-09T07:38:58.301 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /etc/ceph 2023-10-09T07:38:58.332 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 777 /etc/ceph 2023-10-09T07:38:58.399 INFO:tasks.cephadm:Writing seed config... 2023-10-09T07:38:58.400 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-10-09T07:38:58.400 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-10-09T07:38:58.400 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-10-09T07:38:58.400 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-10-09T07:38:58.400 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-10-09T07:38:58.400 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-10-09T07:38:58.401 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-10-09T07:38:58.401 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-10-09T07:38:58.401 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-10-09T07:38:58.457 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = d56ccb5c-6676-11ee-8db5-212e2dc638e7 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2023-10-09T07:38:58.457 DEBUG:teuthology.orchestra.run.smithi115:mon.a> sudo journalctl -f -n 0 -u ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@mon.a.service 2023-10-09T07:38:58.499 DEBUG:teuthology.orchestra.run.smithi115:mgr.x> sudo journalctl -f -n 0 -u ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@mgr.x.service 2023-10-09T07:38:58.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:-- Logs begin at Mon 2023-10-09 07:30:34 UTC. -- 2023-10-09T07:38:58.527 INFO:tasks.cephadm:Bootstrapping... 2023-10-09T07:38:58.527 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 -v bootstrap --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.115 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-10-09T07:38:58.552 INFO:journalctl@ceph.mgr.x.smithi115.stdout:-- Logs begin at Mon 2023-10-09 07:30:34 UTC. -- 2023-10-09T07:38:58.686 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-10-09T07:38:58.686 INFO:teuthology.orchestra.run.smithi115.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2', '-v', 'bootstrap', '--fsid', 'd56ccb5c-6676-11ee-8db5-212e2dc638e7', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.115', '--skip-admin-label'] 2023-10-09T07:38:58.717 INFO:teuthology.orchestra.run.smithi115.stdout:/bin/podman: stdout 4.6.1 2023-10-09T07:38:58.717 INFO:teuthology.orchestra.run.smithi115.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-10-09T07:38:58.717 INFO:teuthology.orchestra.run.smithi115.stdout:Verifying podman|docker is present... 2023-10-09T07:38:58.747 INFO:teuthology.orchestra.run.smithi115.stdout:/bin/podman: stdout 4.6.1 2023-10-09T07:38:58.747 INFO:teuthology.orchestra.run.smithi115.stdout:Verifying lvm2 is present... 2023-10-09T07:38:58.747 INFO:teuthology.orchestra.run.smithi115.stdout:Verifying time synchronization is in place... 2023-10-09T07:38:58.753 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-10-09T07:38:58.754 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-10-09T07:38:58.759 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-10-09T07:38:58.759 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout inactive 2023-10-09T07:38:58.765 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout enabled 2023-10-09T07:38:58.771 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout active 2023-10-09T07:38:58.771 INFO:teuthology.orchestra.run.smithi115.stdout:Unit chronyd.service is enabled and running 2023-10-09T07:38:58.771 INFO:teuthology.orchestra.run.smithi115.stdout:Repeating the final host check... 2023-10-09T07:38:58.803 INFO:teuthology.orchestra.run.smithi115.stdout:/bin/podman: stdout 4.6.1 2023-10-09T07:38:58.803 INFO:teuthology.orchestra.run.smithi115.stdout:podman (/bin/podman) version 4.6.1 is present 2023-10-09T07:38:58.803 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl is present 2023-10-09T07:38:58.803 INFO:teuthology.orchestra.run.smithi115.stdout:lvcreate is present 2023-10-09T07:38:58.809 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-10-09T07:38:58.809 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-10-09T07:38:58.816 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-10-09T07:38:58.816 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout inactive 2023-10-09T07:38:58.821 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout enabled 2023-10-09T07:38:58.827 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout active 2023-10-09T07:38:58.827 INFO:teuthology.orchestra.run.smithi115.stdout:Unit chronyd.service is enabled and running 2023-10-09T07:38:58.827 INFO:teuthology.orchestra.run.smithi115.stdout:Host looks OK 2023-10-09T07:38:58.828 INFO:teuthology.orchestra.run.smithi115.stdout:Cluster fsid: d56ccb5c-6676-11ee-8db5-212e2dc638e7 2023-10-09T07:38:58.828 INFO:teuthology.orchestra.run.smithi115.stdout:Acquiring lock 140229716626848 on /run/cephadm/d56ccb5c-6676-11ee-8db5-212e2dc638e7.lock 2023-10-09T07:38:58.828 INFO:teuthology.orchestra.run.smithi115.stdout:Lock 140229716626848 acquired on /run/cephadm/d56ccb5c-6676-11ee-8db5-212e2dc638e7.lock 2023-10-09T07:38:58.829 INFO:teuthology.orchestra.run.smithi115.stdout:Verifying IP 172.21.15.115 port 3300 ... 2023-10-09T07:38:58.829 INFO:teuthology.orchestra.run.smithi115.stdout:Verifying IP 172.21.15.115 port 6789 ... 2023-10-09T07:38:58.829 INFO:teuthology.orchestra.run.smithi115.stdout:Base mon IP(s) is [172.21.15.115:3300, 172.21.15.115:6789], mon addrv is [v2:172.21.15.115:3300,v1:172.21.15.115:6789] 2023-10-09T07:38:58.832 INFO:teuthology.orchestra.run.smithi115.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.115 metric 100 2023-10-09T07:38:58.832 INFO:teuthology.orchestra.run.smithi115.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.115 metric 100 2023-10-09T07:38:58.834 INFO:teuthology.orchestra.run.smithi115.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-10-09T07:38:58.835 INFO:teuthology.orchestra.run.smithi115.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2023-10-09T07:38:58.835 INFO:teuthology.orchestra.run.smithi115.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1796sec hoplimit 64 pref medium 2023-10-09T07:38:58.837 INFO:teuthology.orchestra.run.smithi115.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-10-09T07:38:58.837 INFO:teuthology.orchestra.run.smithi115.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-10-09T07:38:58.837 INFO:teuthology.orchestra.run.smithi115.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-10-09T07:38:58.837 INFO:teuthology.orchestra.run.smithi115.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-10-09T07:38:58.837 INFO:teuthology.orchestra.run.smithi115.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:ce8f/64 scope link 2023-10-09T07:38:58.838 INFO:teuthology.orchestra.run.smithi115.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-10-09T07:38:58.838 INFO:teuthology.orchestra.run.smithi115.stdout:Mon IP `172.21.15.115` is in CIDR network `172.21.0.0/20` 2023-10-09T07:38:58.838 INFO:teuthology.orchestra.run.smithi115.stdout:Mon IP `172.21.15.115` is in CIDR network `172.21.0.0/20` 2023-10-09T07:38:58.838 INFO:teuthology.orchestra.run.smithi115.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-10-09T07:38:58.839 INFO:teuthology.orchestra.run.smithi115.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-10-09T07:38:58.839 INFO:teuthology.orchestra.run.smithi115.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2... 2023-10-09T07:38:59.221 INFO:teuthology.orchestra.run.smithi115.stdout:/bin/podman: stdout fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863 2023-10-09T07:38:59.221 INFO:teuthology.orchestra.run.smithi115.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2... 2023-10-09T07:38:59.221 INFO:teuthology.orchestra.run.smithi115.stdout:/bin/podman: stderr Getting image source signatures 2023-10-09T07:38:59.221 INFO:teuthology.orchestra.run.smithi115.stdout:/bin/podman: stderr Copying blob sha256:556fc22cf1e739705d2013195476c145689a6e17c5a9071e0d49fdc6e036642a 2023-10-09T07:38:59.221 INFO:teuthology.orchestra.run.smithi115.stdout:/bin/podman: stderr Copying blob sha256:3f4fd5c3c800c6da9ef94691df203a05a5bbbf32c483d09d620ec06eef42ea2b 2023-10-09T07:38:59.221 INFO:teuthology.orchestra.run.smithi115.stdout:/bin/podman: stderr Copying config sha256:fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863 2023-10-09T07:38:59.221 INFO:teuthology.orchestra.run.smithi115.stdout:/bin/podman: stderr Writing manifest to image destination 2023-10-09T07:38:59.798 INFO:teuthology.orchestra.run.smithi115.stdout:ceph: stdout ceph version 18.0.0-5817-gbcea8c48 (bcea8c48517ff154f392584022efa0718f51c9f2) reef (dev) 2023-10-09T07:38:59.798 INFO:teuthology.orchestra.run.smithi115.stdout:Ceph version: ceph version 18.0.0-5817-gbcea8c48 (bcea8c48517ff154f392584022efa0718f51c9f2) reef (dev) 2023-10-09T07:38:59.798 INFO:teuthology.orchestra.run.smithi115.stdout:Extracting ceph user uid/gid from container image... 2023-10-09T07:39:00.310 INFO:teuthology.orchestra.run.smithi115.stdout:stat: stdout 167 167 2023-10-09T07:39:00.310 INFO:teuthology.orchestra.run.smithi115.stdout:Creating initial keys... 2023-10-09T07:39:00.814 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph-authtool: stdout AQAUriNlPh9kHBAAmmEGDoU99eObZV/WMeyY2w== 2023-10-09T07:39:01.632 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph-authtool: stdout AQAUriNlfE4xOhAA2krqmB47N56tpu+dSTD6CA== 2023-10-09T07:39:02.119 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph-authtool: stdout AQAVriNlod3uLxAA77gIa7WvXD+KQhcirIxNhg== 2023-10-09T07:39:02.119 INFO:teuthology.orchestra.run.smithi115.stdout:Creating initial monmap... 2023-10-09T07:39:02.630 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-10-09T07:39:02.630 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-10-09T07:39:02.630 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to d56ccb5c-6676-11ee-8db5-212e2dc638e7 2023-10-09T07:39:02.631 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-10-09T07:39:02.631 INFO:teuthology.orchestra.run.smithi115.stdout:monmaptool for a [v2:172.21.15.115:3300,v1:172.21.15.115:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-10-09T07:39:02.631 INFO:teuthology.orchestra.run.smithi115.stdout:setting min_mon_release = pacific 2023-10-09T07:39:02.631 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/monmaptool: set fsid to d56ccb5c-6676-11ee-8db5-212e2dc638e7 2023-10-09T07:39:02.631 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-10-09T07:39:02.631 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:39:02.631 INFO:teuthology.orchestra.run.smithi115.stdout:Creating mon... 2023-10-09T07:39:03.159 INFO:teuthology.orchestra.run.smithi115.stdout:create mon.a on 2023-10-09T07:39:03.377 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-10-09T07:39:03.502 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7.target → /etc/systemd/system/ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7.target. 2023-10-09T07:39:03.503 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7.target → /etc/systemd/system/ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7.target. 2023-10-09T07:39:03.681 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@mon.a 2023-10-09T07:39:03.681 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@mon.a.service: Unit ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@mon.a.service not loaded. 2023-10-09T07:39:03.769 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7.target.wants/ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@mon.a.service → /etc/systemd/system/ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@.service. 2023-10-09T07:39:04.098 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-10-09T07:39:04.098 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout disabled 2023-10-09T07:39:04.106 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-10-09T07:39:04.106 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout inactive 2023-10-09T07:39:04.106 INFO:teuthology.orchestra.run.smithi115.stdout:firewalld.service is not enabled 2023-10-09T07:39:04.106 INFO:teuthology.orchestra.run.smithi115.stdout:Not possible to enable service . firewalld.service is not available 2023-10-09T07:39:04.107 INFO:teuthology.orchestra.run.smithi115.stdout:Waiting for mon to start... 2023-10-09T07:39:04.107 INFO:teuthology.orchestra.run.smithi115.stdout:Waiting for mon... 2023-10-09T07:39:04.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:04 smithi115 ceph-mon[107634]: mon.a is new leader, mons a in quorum (ranks 0) 2023-10-09T07:39:04.804 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout cluster: 2023-10-09T07:39:04.805 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout id: d56ccb5c-6676-11ee-8db5-212e2dc638e7 2023-10-09T07:39:04.805 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-10-09T07:39:04.805 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-10-09T07:39:04.805 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout services: 2023-10-09T07:39:04.805 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.363683s) 2023-10-09T07:39:04.805 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-10-09T07:39:04.805 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-10-09T07:39:04.805 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-10-09T07:39:04.805 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout data: 2023-10-09T07:39:04.805 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-10-09T07:39:04.805 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-10-09T07:39:04.806 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-10-09T07:39:04.806 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout pgs: 2023-10-09T07:39:04.806 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-10-09T07:39:04.806 INFO:teuthology.orchestra.run.smithi115.stdout:mon is available 2023-10-09T07:39:04.806 INFO:teuthology.orchestra.run.smithi115.stdout:Assimilating anything we can from ceph.conf... 2023-10-09T07:39:05.505 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:05 smithi115 ceph-mon[107634]: mon.a is new leader, mons a in quorum (ranks 0) 2023-10-09T07:39:05.505 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:05 smithi115 ceph-mon[107634]: monmap e1: 1 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0]} removed_ranks: {} 2023-10-09T07:39:05.505 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:05 smithi115 ceph-mon[107634]: fsmap 2023-10-09T07:39:05.505 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:05 smithi115 ceph-mon[107634]: osdmap e1: 0 total, 0 up, 0 in 2023-10-09T07:39:05.505 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:05 smithi115 ceph-mon[107634]: mgrmap e1: no daemons active 2023-10-09T07:39:05.505 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:05 smithi115 ceph-mon[107634]: from='client.? 172.21.15.115:0/1574438381' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-10-09T07:39:05.561 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-10-09T07:39:05.562 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout [global] 2023-10-09T07:39:05.562 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout fsid = d56ccb5c-6676-11ee-8db5-212e2dc638e7 2023-10-09T07:39:05.562 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.115:3300,v1:172.21.15.115:6789] 2023-10-09T07:39:05.562 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-10-09T07:39:05.562 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-10-09T07:39:05.562 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-10-09T07:39:05.562 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-10-09T07:39:05.562 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-10-09T07:39:05.563 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout [mgr] 2023-10-09T07:39:05.563 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-10-09T07:39:05.563 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-10-09T07:39:05.563 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout [osd] 2023-10-09T07:39:05.563 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-10-09T07:39:05.563 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-10-09T07:39:05.563 INFO:teuthology.orchestra.run.smithi115.stdout:Generating new minimal ceph.conf... 2023-10-09T07:39:06.286 INFO:teuthology.orchestra.run.smithi115.stdout:Restarting the monitor... 2023-10-09T07:39:06.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:06 smithi115 ceph-mon[107634]: from='client.? 172.21.15.115:0/855056486' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-10-09T07:39:06.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:06 smithi115 ceph-mon[107634]: from='client.? 172.21.15.115:0/855056486' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-10-09T07:39:06.293 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:06 smithi115 ceph-mon[107634]: from='client.? 172.21.15.115:0/2817049010' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:39:06.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:06 smithi115 systemd[1]: Stopping Ceph mon.a for d56ccb5c-6676-11ee-8db5-212e2dc638e7... 2023-10-09T07:39:06.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:06 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mon-a[107630]: 2023-10-09T07:39:06.453+0000 7f6d7b5ec700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-10-09T07:39:06.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:06 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mon-a[107630]: 2023-10-09T07:39:06.453+0000 7f6d7b5ec700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-10-09T07:39:07.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:06 smithi115 bash[107885]: ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mon-a 2023-10-09T07:39:07.409 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 systemd[1]: ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@mon.a.service: Succeeded. 2023-10-09T07:39:07.409 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 systemd[1]: Stopped Ceph mon.a for d56ccb5c-6676-11ee-8db5-212e2dc638e7. 2023-10-09T07:39:07.409 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 systemd[1]: Starting Ceph mon.a for d56ccb5c-6676-11ee-8db5-212e2dc638e7... 2023-10-09T07:39:07.412 INFO:teuthology.orchestra.run.smithi115.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2023-10-09T07:39:07.787 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 bash[107996]: 020fdc55563b7483f61b0d9e5894dbc5d31a58c90363e8d9d1b652ff6fea72b6 2023-10-09T07:39:07.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 systemd[1]: Started Ceph mon.a for d56ccb5c-6676-11ee-8db5-212e2dc638e7. 2023-10-09T07:39:07.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: set uid:gid to 167:167 (ceph:ceph) 2023-10-09T07:39:07.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: ceph version 18.0.0-5817-gbcea8c48 (bcea8c48517ff154f392584022efa0718f51c9f2) reef (dev), process ceph-mon, pid 2 2023-10-09T07:39:07.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: pidfile_write: ignore empty --pid-file 2023-10-09T07:39:07.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: load: jerasure load: lrc 2023-10-09T07:39:07.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: RocksDB version: 7.9.2 2023-10-09T07:39:07.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Git sha 0 2023-10-09T07:39:07.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Compile date 2023-09-25 19:01:54 2023-10-09T07:39:07.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: DB SUMMARY 2023-10-09T07:39:07.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: DB Session ID: 148TBMV3E2VASS4DUK0M 2023-10-09T07:39:07.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: CURRENT file: CURRENT 2023-10-09T07:39:07.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: IDENTITY file: IDENTITY 2023-10-09T07:39:07.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-10-09T07:39:07.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-10-09T07:39:07.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 82842 ; 2023-10-09T07:39:07.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.error_if_exists: 0 2023-10-09T07:39:07.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.create_if_missing: 0 2023-10-09T07:39:07.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.paranoid_checks: 1 2023-10-09T07:39:07.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.flush_verify_memtable_count: 1 2023-10-09T07:39:07.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-10-09T07:39:07.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-10-09T07:39:07.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.env: 0x563529ae3da0 2023-10-09T07:39:07.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.fs: PosixFileSystem 2023-10-09T07:39:07.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.info_log: 0x56352a5f7460 2023-10-09T07:39:07.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_file_opening_threads: 16 2023-10-09T07:39:07.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.statistics: (nil) 2023-10-09T07:39:07.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.use_fsync: 0 2023-10-09T07:39:07.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_log_file_size: 0 2023-10-09T07:39:07.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-10-09T07:39:07.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.log_file_time_to_roll: 0 2023-10-09T07:39:07.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.keep_log_file_num: 1000 2023-10-09T07:39:07.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.recycle_log_file_num: 0 2023-10-09T07:39:07.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.allow_fallocate: 1 2023-10-09T07:39:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.allow_mmap_reads: 0 2023-10-09T07:39:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.allow_mmap_writes: 0 2023-10-09T07:39:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.use_direct_reads: 0 2023-10-09T07:39:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-10-09T07:39:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.create_missing_column_families: 0 2023-10-09T07:39:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.db_log_dir: 2023-10-09T07:39:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.wal_dir: 2023-10-09T07:39:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.table_cache_numshardbits: 6 2023-10-09T07:39:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.WAL_ttl_seconds: 0 2023-10-09T07:39:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.WAL_size_limit_MB: 0 2023-10-09T07:39:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-10-09T07:39:07.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-10-09T07:39:07.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.is_fd_close_on_exec: 1 2023-10-09T07:39:07.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.advise_random_on_open: 1 2023-10-09T07:39:07.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.db_write_buffer_size: 0 2023-10-09T07:39:07.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.write_buffer_manager: 0x56352a5b14a0 2023-10-09T07:39:07.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-10-09T07:39:07.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-10-09T07:39:07.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.use_adaptive_mutex: 0 2023-10-09T07:39:07.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.rate_limiter: (nil) 2023-10-09T07:39:07.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-10-09T07:39:07.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.wal_recovery_mode: 2 2023-10-09T07:39:07.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.enable_thread_tracking: 0 2023-10-09T07:39:07.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.enable_pipelined_write: 0 2023-10-09T07:39:07.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.unordered_write: 0 2023-10-09T07:39:07.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-10-09T07:39:07.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-10-09T07:39:07.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-10-09T07:39:07.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-10-09T07:39:07.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.row_cache: None 2023-10-09T07:39:07.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.wal_filter: None 2023-10-09T07:39:07.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-10-09T07:39:07.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.allow_ingest_behind: 0 2023-10-09T07:39:07.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.two_write_queues: 0 2023-10-09T07:39:07.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.manual_wal_flush: 0 2023-10-09T07:39:07.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.wal_compression: 0 2023-10-09T07:39:07.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.atomic_flush: 0 2023-10-09T07:39:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-10-09T07:39:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.persist_stats_to_disk: 0 2023-10-09T07:39:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.write_dbid_to_manifest: 0 2023-10-09T07:39:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.log_readahead_size: 0 2023-10-09T07:39:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-10-09T07:39:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.best_efforts_recovery: 0 2023-10-09T07:39:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-10-09T07:39:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-10-09T07:39:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.allow_data_in_errors: 0 2023-10-09T07:39:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.db_host_id: __hostname__ 2023-10-09T07:39:07.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.enforce_single_del_contracts: true 2023-10-09T07:39:07.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_background_jobs: 2 2023-10-09T07:39:07.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_background_compactions: -1 2023-10-09T07:39:07.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_subcompactions: 1 2023-10-09T07:39:07.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-10-09T07:39:07.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-10-09T07:39:07.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.delayed_write_rate : 16777216 2023-10-09T07:39:07.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_total_wal_size: 0 2023-10-09T07:39:07.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-10-09T07:39:07.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.stats_dump_period_sec: 600 2023-10-09T07:39:07.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.stats_persist_period_sec: 600 2023-10-09T07:39:07.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-10-09T07:39:07.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_open_files: -1 2023-10-09T07:39:07.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.bytes_per_sync: 0 2023-10-09T07:39:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.wal_bytes_per_sync: 0 2023-10-09T07:39:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.strict_bytes_per_sync: 0 2023-10-09T07:39:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.compaction_readahead_size: 0 2023-10-09T07:39:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_background_flushes: -1 2023-10-09T07:39:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Compression algorithms supported: 2023-10-09T07:39:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-10-09T07:39:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: kZSTD supported: 0 2023-10-09T07:39:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: kXpressCompression supported: 0 2023-10-09T07:39:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: kLZ4HCCompression supported: 1 2023-10-09T07:39:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: kZlibCompression supported: 1 2023-10-09T07:39:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: kSnappyCompression supported: 1 2023-10-09T07:39:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: kLZ4Compression supported: 1 2023-10-09T07:39:07.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: kBZip2Compression supported: 0 2023-10-09T07:39:07.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Fast CRC32 supported: Supported on x86 2023-10-09T07:39:07.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: DMutex implementation: pthread_mutex_t 2023-10-09T07:39:07.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-10-09T07:39:07.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-10-09T07:39:07.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-10-09T07:39:07.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.merge_operator: 2023-10-09T07:39:07.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.compaction_filter: None 2023-10-09T07:39:07.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.compaction_filter_factory: None 2023-10-09T07:39:07.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.sst_partitioner_factory: None 2023-10-09T07:39:07.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.memtable_factory: SkipListFactory 2023-10-09T07:39:07.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.table_factory: BlockBasedTable 2023-10-09T07:39:07.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56352a5f7560) 2023-10-09T07:39:07.798 INFO:journalctl@ceph.mon.a.smithi115.stdout: cache_index_and_filter_blocks: 1 2023-10-09T07:39:07.798 INFO:journalctl@ceph.mon.a.smithi115.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-10-09T07:39:07.798 INFO:journalctl@ceph.mon.a.smithi115.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-10-09T07:39:07.798 INFO:journalctl@ceph.mon.a.smithi115.stdout: pin_top_level_index_and_filter: 1 2023-10-09T07:39:07.798 INFO:journalctl@ceph.mon.a.smithi115.stdout: index_type: 0 2023-10-09T07:39:07.798 INFO:journalctl@ceph.mon.a.smithi115.stdout: data_block_index_type: 0 2023-10-09T07:39:07.798 INFO:journalctl@ceph.mon.a.smithi115.stdout: index_shortening: 1 2023-10-09T07:39:07.798 INFO:journalctl@ceph.mon.a.smithi115.stdout: data_block_hash_table_util_ratio: 0.750000 2023-10-09T07:39:07.798 INFO:journalctl@ceph.mon.a.smithi115.stdout: checksum: 4 2023-10-09T07:39:07.798 INFO:journalctl@ceph.mon.a.smithi115.stdout: no_block_cache: 0 2023-10-09T07:39:07.798 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_cache: 0x56352a64f1f0 2023-10-09T07:39:07.798 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_cache_name: BinnedLRUCache 2023-10-09T07:39:07.799 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_cache_options: 2023-10-09T07:39:07.799 INFO:journalctl@ceph.mon.a.smithi115.stdout: capacity : 536870912 2023-10-09T07:39:07.799 INFO:journalctl@ceph.mon.a.smithi115.stdout: num_shard_bits : 4 2023-10-09T07:39:07.799 INFO:journalctl@ceph.mon.a.smithi115.stdout: strict_capacity_limit : 0 2023-10-09T07:39:07.799 INFO:journalctl@ceph.mon.a.smithi115.stdout: high_pri_pool_ratio: 0.000 2023-10-09T07:39:07.799 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_cache_compressed: (nil) 2023-10-09T07:39:07.799 INFO:journalctl@ceph.mon.a.smithi115.stdout: persistent_cache: (nil) 2023-10-09T07:39:07.799 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_size: 4096 2023-10-09T07:39:07.799 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_size_deviation: 10 2023-10-09T07:39:07.799 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_restart_interval: 16 2023-10-09T07:39:07.799 INFO:journalctl@ceph.mon.a.smithi115.stdout: index_block_restart_interval: 1 2023-10-09T07:39:07.799 INFO:journalctl@ceph.mon.a.smithi115.stdout: metadata_block_size: 4096 2023-10-09T07:39:07.799 INFO:journalctl@ceph.mon.a.smithi115.stdout: partition_filters: 0 2023-10-09T07:39:07.800 INFO:journalctl@ceph.mon.a.smithi115.stdout: use_delta_encoding: 1 2023-10-09T07:39:07.800 INFO:journalctl@ceph.mon.a.smithi115.stdout: filter_policy: bloomfilter 2023-10-09T07:39:07.800 INFO:journalctl@ceph.mon.a.smithi115.stdout: whole_key_filtering: 1 2023-10-09T07:39:07.800 INFO:journalctl@ceph.mon.a.smithi115.stdout: verify_compression: 0 2023-10-09T07:39:07.800 INFO:journalctl@ceph.mon.a.smithi115.stdout: read_amp_bytes_per_bit: 0 2023-10-09T07:39:07.800 INFO:journalctl@ceph.mon.a.smithi115.stdout: format_version: 5 2023-10-09T07:39:07.800 INFO:journalctl@ceph.mon.a.smithi115.stdout: enable_index_compression: 1 2023-10-09T07:39:07.800 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_align: 0 2023-10-09T07:39:07.800 INFO:journalctl@ceph.mon.a.smithi115.stdout: max_auto_readahead_size: 262144 2023-10-09T07:39:07.800 INFO:journalctl@ceph.mon.a.smithi115.stdout: prepopulate_block_cache: 0 2023-10-09T07:39:07.800 INFO:journalctl@ceph.mon.a.smithi115.stdout: initial_auto_readahead_size: 8192 2023-10-09T07:39:07.800 INFO:journalctl@ceph.mon.a.smithi115.stdout: num_file_reads_for_auto_readahead: 2 2023-10-09T07:39:07.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.write_buffer_size: 33554432 2023-10-09T07:39:07.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_write_buffer_number: 2 2023-10-09T07:39:07.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.compression: NoCompression 2023-10-09T07:39:07.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.bottommost_compression: Disabled 2023-10-09T07:39:07.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.prefix_extractor: nullptr 2023-10-09T07:39:07.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-10-09T07:39:07.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.num_levels: 7 2023-10-09T07:39:07.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-10-09T07:39:07.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-10-09T07:39:07.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-10-09T07:39:07.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-10-09T07:39:07.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-10-09T07:39:07.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-10-09T07:39:07.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-10-09T07:39:07.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-10-09T07:39:07.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-10-09T07:39:07.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-10-09T07:39:07.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-10-09T07:39:07.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-10-09T07:39:07.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.compression_opts.window_bits: -14 2023-10-09T07:39:07.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.compression_opts.level: 32767 2023-10-09T07:39:07.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.compression_opts.strategy: 0 2023-10-09T07:39:07.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-10-09T07:39:07.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-10-09T07:39:07.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-10-09T07:39:07.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-10-09T07:39:07.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.compression_opts.enabled: false 2023-10-09T07:39:07.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-10-09T07:39:07.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-10-09T07:39:07.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-10-09T07:39:07.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-10-09T07:39:07.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.target_file_size_base: 67108864 2023-10-09T07:39:07.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.target_file_size_multiplier: 1 2023-10-09T07:39:07.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-10-09T07:39:07.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-10-09T07:39:07.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-10-09T07:39:07.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-10-09T07:39:07.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-10-09T07:39:07.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-10-09T07:39:07.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-10-09T07:39:07.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-10-09T07:39:07.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-10-09T07:39:07.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-10-09T07:39:07.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-10-09T07:39:07.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-10-09T07:39:07.805 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-10-09T07:39:07.805 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.arena_block_size: 1048576 2023-10-09T07:39:07.805 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-10-09T07:39:07.805 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-10-09T07:39:07.805 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.disable_auto_compactions: 0 2023-10-09T07:39:07.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-10-09T07:39:07.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-10-09T07:39:07.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-10-09T07:39:07.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-10-09T07:39:07.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-10-09T07:39:07.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-10-09T07:39:07.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-10-09T07:39:07.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-10-09T07:39:07.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-10-09T07:39:07.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-10-09T07:39:07.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.table_properties_collectors: 2023-10-09T07:39:07.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.inplace_update_support: 0 2023-10-09T07:39:07.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.inplace_update_num_locks: 10000 2023-10-09T07:39:07.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-10-09T07:39:07.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-10-09T07:39:07.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.memtable_huge_page_size: 0 2023-10-09T07:39:07.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.bloom_locality: 0 2023-10-09T07:39:07.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.max_successive_merges: 0 2023-10-09T07:39:07.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.optimize_filters_for_hits: 0 2023-10-09T07:39:07.808 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.paranoid_file_checks: 0 2023-10-09T07:39:07.808 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.force_consistency_checks: 1 2023-10-09T07:39:07.808 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.report_bg_io_stats: 0 2023-10-09T07:39:07.808 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.ttl: 2592000 2023-10-09T07:39:07.808 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.periodic_compaction_seconds: 0 2023-10-09T07:39:07.808 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-10-09T07:39:07.808 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-10-09T07:39:07.808 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.enable_blob_files: false 2023-10-09T07:39:07.808 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.min_blob_size: 0 2023-10-09T07:39:07.808 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.blob_file_size: 268435456 2023-10-09T07:39:07.808 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.blob_compression_type: NoCompression 2023-10-09T07:39:07.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.enable_blob_garbage_collection: false 2023-10-09T07:39:07.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-10-09T07:39:07.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-10-09T07:39:07.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-10-09T07:39:07.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.blob_file_starting_level: 0 2023-10-09T07:39:07.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-10-09T07:39:07.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-10-09T07:39:07.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-10-09T07:39:07.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d2559596-da65-4960-a28d-3e1696280aaa 2023-10-09T07:39:07.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: EVENT_LOG_v1 {"time_micros": 1696837147425823, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-10-09T07:39:07.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-10-09T07:39:07.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: EVENT_LOG_v1 {"time_micros": 1696837147426670, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 79767, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 242, "table_properties": {"data_size": 77947, "index_size": 230, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9724, "raw_average_key_size": 47, "raw_value_size": 72390, "raw_average_value_size": 353, "num_data_blocks": 10, "num_entries": 205, "num_filter_entries": 205, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1696837147, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d2559596-da65-4960-a28d-3e1696280aaa", "db_session_id": "148TBMV3E2VASS4DUK0M", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-10-09T07:39:07.810 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: EVENT_LOG_v1 {"time_micros": 1696837147426750, "job": 1, "event": "recovery_finished"} 2023-10-09T07:39:07.810 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-10-09T07:39:07.810 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-10-09T07:39:07.810 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56352a6ec000 2023-10-09T07:39:07.810 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: DB pointer 0x56352a6d8000 2023-10-09T07:39:07.810 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-10-09T07:39:07.810 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-10-09T07:39:07.810 INFO:journalctl@ceph.mon.a.smithi115.stdout: ** DB Stats ** 2023-10-09T07:39:07.810 INFO:journalctl@ceph.mon.a.smithi115.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-10-09T07:39:07.810 INFO:journalctl@ceph.mon.a.smithi115.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-10-09T07:39:07.810 INFO:journalctl@ceph.mon.a.smithi115.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-10-09T07:39:07.810 INFO:journalctl@ceph.mon.a.smithi115.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-09T07:39:07.810 INFO:journalctl@ceph.mon.a.smithi115.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-10-09T07:39:07.811 INFO:journalctl@ceph.mon.a.smithi115.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-10-09T07:39:07.811 INFO:journalctl@ceph.mon.a.smithi115.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-09T07:39:07.811 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2023-10-09T07:39:07.811 INFO:journalctl@ceph.mon.a.smithi115.stdout: ** Compaction Stats [default] ** 2023-10-09T07:39:07.811 INFO:journalctl@ceph.mon.a.smithi115.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-10-09T07:39:07.811 INFO:journalctl@ceph.mon.a.smithi115.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-10-09T07:39:07.811 INFO:journalctl@ceph.mon.a.smithi115.stdout: L0 2/0 79.73 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-10-09T07:39:07.811 INFO:journalctl@ceph.mon.a.smithi115.stdout: Sum 2/0 79.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-10-09T07:39:07.811 INFO:journalctl@ceph.mon.a.smithi115.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-10-09T07:39:07.811 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2023-10-09T07:39:07.811 INFO:journalctl@ceph.mon.a.smithi115.stdout: ** Compaction Stats [default] ** 2023-10-09T07:39:07.811 INFO:journalctl@ceph.mon.a.smithi115.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-10-09T07:39:07.812 INFO:journalctl@ceph.mon.a.smithi115.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-10-09T07:39:07.812 INFO:journalctl@ceph.mon.a.smithi115.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 141.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-10-09T07:39:07.812 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2023-10-09T07:39:07.812 INFO:journalctl@ceph.mon.a.smithi115.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-10-09T07:39:07.812 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2023-10-09T07:39:07.812 INFO:journalctl@ceph.mon.a.smithi115.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-10-09T07:39:07.812 INFO:journalctl@ceph.mon.a.smithi115.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-10-09T07:39:07.812 INFO:journalctl@ceph.mon.a.smithi115.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-10-09T07:39:07.812 INFO:journalctl@ceph.mon.a.smithi115.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-10-09T07:39:07.812 INFO:journalctl@ceph.mon.a.smithi115.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-10-09T07:39:07.812 INFO:journalctl@ceph.mon.a.smithi115.stdout: AddFile(Keys): cumulative 0, interval 0 2023-10-09T07:39:07.812 INFO:journalctl@ceph.mon.a.smithi115.stdout: Cumulative compaction: 0.00 GB write, 17.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-09T07:39:07.812 INFO:journalctl@ceph.mon.a.smithi115.stdout: Interval compaction: 0.00 GB write, 17.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-09T07:39:07.813 INFO:journalctl@ceph.mon.a.smithi115.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-10-09T07:39:07.813 INFO:journalctl@ceph.mon.a.smithi115.stdout: Block cache BinnedLRUCache@0x56352a64f1f0#2 capacity: 512.00 MB usage: 1.19 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2023-10-09T07:39:07.813 INFO:journalctl@ceph.mon.a.smithi115.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.42 KB,8.04663e-05%) Misc(1,0.00 KB,0%) 2023-10-09T07:39:07.813 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2023-10-09T07:39:07.813 INFO:journalctl@ceph.mon.a.smithi115.stdout: ** File Read Latency Histogram By Level [default] ** 2023-10-09T07:39:07.813 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: starting mon.a rank 0 at public addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] at bind addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 2023-10-09T07:39:07.813 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: mon.a@-1(???) e1 preinit fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 2023-10-09T07:39:07.813 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: mon.a@-1(???).mds e1 new map 2023-10-09T07:39:07.813 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: mon.a@-1(???).mds e1 print_map 2023-10-09T07:39:07.813 INFO:journalctl@ceph.mon.a.smithi115.stdout: e1 2023-10-09T07:39:07.813 INFO:journalctl@ceph.mon.a.smithi115.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-10-09T07:39:07.814 INFO:journalctl@ceph.mon.a.smithi115.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2023-10-09T07:39:07.814 INFO:journalctl@ceph.mon.a.smithi115.stdout: legacy client fscid: -1 2023-10-09T07:39:07.814 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2023-10-09T07:39:07.814 INFO:journalctl@ceph.mon.a.smithi115.stdout: No filesystems configured 2023-10-09T07:39:07.814 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-10-09T07:39:07.814 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-10-09T07:39:07.814 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-10-09T07:39:07.814 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-10-09T07:39:07.814 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-10-09T07:39:07.814 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: expand_channel_meta expand map: {default=false} 2023-10-09T07:39:07.814 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: expand_channel_meta from 'false' to 'false' 2023-10-09T07:39:07.814 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: expand_channel_meta expanded map: {default=false} 2023-10-09T07:39:07.815 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: expand_channel_meta expand map: {default=info} 2023-10-09T07:39:07.815 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: expand_channel_meta from 'info' to 'info' 2023-10-09T07:39:07.815 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: expand_channel_meta expanded map: {default=info} 2023-10-09T07:39:07.815 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: expand_channel_meta expand map: {default=daemon} 2023-10-09T07:39:07.815 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: expand_channel_meta from 'daemon' to 'daemon' 2023-10-09T07:39:07.815 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: expand_channel_meta expanded map: {default=daemon} 2023-10-09T07:39:07.815 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: expand_channel_meta expand map: {default=debug} 2023-10-09T07:39:07.815 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: expand_channel_meta from 'debug' to 'debug' 2023-10-09T07:39:07.815 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: expand_channel_meta expanded map: {default=debug} 2023-10-09T07:39:07.815 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: mon.a is new leader, mons a in quorum (ranks 0) 2023-10-09T07:39:07.815 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: monmap e1: 1 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0]} removed_ranks: {} 2023-10-09T07:39:07.815 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: fsmap 2023-10-09T07:39:07.815 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: osdmap e1: 0 total, 0 up, 0 in 2023-10-09T07:39:07.816 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:07 smithi115 ceph-mon[108015]: mgrmap e1: no daemons active 2023-10-09T07:39:08.128 INFO:teuthology.orchestra.run.smithi115.stdout:Wrote config to /etc/ceph/ceph.conf 2023-10-09T07:39:08.152 INFO:teuthology.orchestra.run.smithi115.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-10-09T07:39:08.152 INFO:teuthology.orchestra.run.smithi115.stdout:Creating mgr... 2023-10-09T07:39:08.153 INFO:teuthology.orchestra.run.smithi115.stdout:Verifying port 0.0.0.0:9283 ... 2023-10-09T07:39:08.153 INFO:teuthology.orchestra.run.smithi115.stdout:Verifying port 0.0.0.0:8765 ... 2023-10-09T07:39:08.324 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@mgr.x 2023-10-09T07:39:08.324 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@mgr.x.service: Unit ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@mgr.x.service not loaded. 2023-10-09T07:39:08.414 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7.target.wants/ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@mgr.x.service → /etc/systemd/system/ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@.service. 2023-10-09T07:39:08.602 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:08 smithi115 systemd[1]: Starting Ceph mgr.x for d56ccb5c-6676-11ee-8db5-212e2dc638e7... 2023-10-09T07:39:08.747 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-10-09T07:39:08.748 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout disabled 2023-10-09T07:39:08.755 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-10-09T07:39:08.756 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout inactive 2023-10-09T07:39:08.756 INFO:teuthology.orchestra.run.smithi115.stdout:firewalld.service is not enabled 2023-10-09T07:39:08.756 INFO:teuthology.orchestra.run.smithi115.stdout:Not possible to enable service . firewalld.service is not available 2023-10-09T07:39:08.764 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-10-09T07:39:08.764 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout disabled 2023-10-09T07:39:08.772 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-10-09T07:39:08.772 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout inactive 2023-10-09T07:39:08.772 INFO:teuthology.orchestra.run.smithi115.stdout:firewalld.service is not enabled 2023-10-09T07:39:08.772 INFO:teuthology.orchestra.run.smithi115.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-10-09T07:39:08.772 INFO:teuthology.orchestra.run.smithi115.stdout:Waiting for mgr to start... 2023-10-09T07:39:08.773 INFO:teuthology.orchestra.run.smithi115.stdout:Waiting for mgr... 2023-10-09T07:39:09.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:08 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/1040942886' entity='client.admin' 2023-10-09T07:39:09.060 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:08 smithi115 bash[108226]: 68ca4335fe8db94e17060d20ae1ea431376fa9aa04d2d0945489c132e952fced 2023-10-09T07:39:09.061 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:08 smithi115 systemd[1]: Started Ceph mgr.x for d56ccb5c-6676-11ee-8db5-212e2dc638e7. 2023-10-09T07:39:09.061 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:08 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:08.878+0000 7f6387e6c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-09T07:39:09.061 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:09 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:09.053+0000 7f6387e6c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-09T07:39:09.555 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-10-09T07:39:09.555 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout { 2023-10-09T07:39:09.555 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "fsid": "d56ccb5c-6676-11ee-8db5-212e2dc638e7", 2023-10-09T07:39:09.555 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "health": { 2023-10-09T07:39:09.555 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-10-09T07:39:09.555 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-10-09T07:39:09.555 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-10-09T07:39:09.555 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:09.555 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-10-09T07:39:09.556 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-10-09T07:39:09.556 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 0 2023-10-09T07:39:09.556 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout ], 2023-10-09T07:39:09.556 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-10-09T07:39:09.556 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "a" 2023-10-09T07:39:09.556 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout ], 2023-10-09T07:39:09.556 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "quorum_age": 1, 2023-10-09T07:39:09.556 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "monmap": { 2023-10-09T07:39:09.556 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-09T07:39:09.556 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-10-09T07:39:09.556 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-10-09T07:39:09.557 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:09.557 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-10-09T07:39:09.557 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-09T07:39:09.557 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-10-09T07:39:09.557 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-10-09T07:39:09.557 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-10-09T07:39:09.557 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-10-09T07:39:09.560 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-10-09T07:39:09.560 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-10-09T07:39:09.561 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:09.561 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-10-09T07:39:09.561 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-10-09T07:39:09.561 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-10-09T07:39:09.561 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-10-09T07:39:09.561 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-10-09T07:39:09.561 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-10-09T07:39:09.561 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-10-09T07:39:09.561 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-10-09T07:39:09.561 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-10-09T07:39:09.561 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:09.561 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-10-09T07:39:09.561 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-09T07:39:09.562 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-10-09T07:39:09.562 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-10-09T07:39:09.562 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:09.562 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-10-09T07:39:09.562 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "available": false, 2023-10-09T07:39:09.562 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-10-09T07:39:09.562 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "modules": [ 2023-10-09T07:39:09.562 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "iostat", 2023-10-09T07:39:09.562 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "nfs", 2023-10-09T07:39:09.562 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "restful" 2023-10-09T07:39:09.562 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout ], 2023-10-09T07:39:09.563 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-09T07:39:09.563 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:09.563 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-10-09T07:39:09.563 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-09T07:39:09.563 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "modified": "2023-10-09T07:39:04.104242+0000", 2023-10-09T07:39:09.563 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-09T07:39:09.563 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:09.563 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-10-09T07:39:09.563 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout } 2023-10-09T07:39:09.563 INFO:teuthology.orchestra.run.smithi115.stdout:mgr not available, waiting (1/15)... 2023-10-09T07:39:09.644 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:09 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:09.392+0000 7f6387e6c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-09T07:39:09.898 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:09 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/1549508186' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-10-09T07:39:10.352 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:09 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:09.897+0000 7f6387e6c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-09T07:39:10.352 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:10 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:10.121+0000 7f6387e6c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-09T07:39:10.852 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:10 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:10.535+0000 7f6387e6c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-09T07:39:11.602 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:11 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:11.242+0000 7f6387e6c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-09T07:39:11.602 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:11 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:11.328+0000 7f6387e6c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-09T07:39:12.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:11 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/676558223' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-10-09T07:39:12.352 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:11 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:11.949+0000 7f6387e6c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-09T07:39:12.352 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:12 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:12.034+0000 7f6387e6c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-09T07:39:12.352 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:12 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:12.122+0000 7f6387e6c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-09T07:39:12.602 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:12 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:12.351+0000 7f6387e6c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-09T07:39:12.602 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:12 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:12.445+0000 7f6387e6c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-09T07:39:12.602 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:12 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:12.537+0000 7f6387e6c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-09T07:39:13.036 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:12 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:12.753+0000 7f6387e6c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-09T07:39:13.036 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:12 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:12.851+0000 7f6387e6c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-09T07:39:13.278 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-10-09T07:39:13.278 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout { 2023-10-09T07:39:13.278 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "fsid": "d56ccb5c-6676-11ee-8db5-212e2dc638e7", 2023-10-09T07:39:13.279 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "health": { 2023-10-09T07:39:13.279 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-10-09T07:39:13.279 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-10-09T07:39:13.279 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-10-09T07:39:13.279 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:13.279 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-10-09T07:39:13.279 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-10-09T07:39:13.279 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 0 2023-10-09T07:39:13.279 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout ], 2023-10-09T07:39:13.279 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-10-09T07:39:13.279 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "a" 2023-10-09T07:39:13.280 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout ], 2023-10-09T07:39:13.280 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2023-10-09T07:39:13.280 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "monmap": { 2023-10-09T07:39:13.280 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-09T07:39:13.280 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-10-09T07:39:13.280 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-10-09T07:39:13.280 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:13.280 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-10-09T07:39:13.280 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-09T07:39:13.280 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-10-09T07:39:13.280 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-10-09T07:39:13.280 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-10-09T07:39:13.281 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-10-09T07:39:13.284 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-10-09T07:39:13.284 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-10-09T07:39:13.284 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:13.284 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-10-09T07:39:13.284 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-10-09T07:39:13.284 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-10-09T07:39:13.284 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-10-09T07:39:13.284 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-10-09T07:39:13.284 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-10-09T07:39:13.285 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-10-09T07:39:13.285 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-10-09T07:39:13.285 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-10-09T07:39:13.285 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:13.285 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-10-09T07:39:13.285 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-09T07:39:13.285 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-10-09T07:39:13.285 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-10-09T07:39:13.285 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:13.285 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-10-09T07:39:13.285 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "available": false, 2023-10-09T07:39:13.285 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-10-09T07:39:13.285 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "modules": [ 2023-10-09T07:39:13.286 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "iostat", 2023-10-09T07:39:13.286 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "nfs", 2023-10-09T07:39:13.286 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "restful" 2023-10-09T07:39:13.286 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout ], 2023-10-09T07:39:13.286 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-09T07:39:13.286 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:13.286 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-10-09T07:39:13.286 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-09T07:39:13.286 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "modified": "2023-10-09T07:39:04.104242+0000", 2023-10-09T07:39:13.286 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-09T07:39:13.286 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:13.286 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-10-09T07:39:13.287 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout } 2023-10-09T07:39:13.287 INFO:teuthology.orchestra.run.smithi115.stdout:mgr not available, waiting (2/15)... 2023-10-09T07:39:13.352 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:13 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:13.035+0000 7f6387e6c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-09T07:39:13.352 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:13 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:13.122+0000 7f6387e6c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-09T07:39:14.265 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:13 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:13.892+0000 7f6387e6c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-09T07:39:14.266 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:14 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:14.087+0000 7f6387e6c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-09T07:39:14.266 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:14 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:14.265+0000 7f6387e6c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-09T07:39:15.281 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:14 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:14.917+0000 7f6387e6c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-09T07:39:15.281 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:15 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:15.097+0000 7f6387e6c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-09T07:39:15.602 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:15 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:15.281+0000 7f6387e6c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-09T07:39:15.602 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:15 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:15.408+0000 7f6387e6c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-09T07:39:15.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:15 smithi115 ceph-mon[108015]: Activating manager daemon x 2023-10-09T07:39:15.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:15 smithi115 ceph-mon[108015]: mgrmap e2: x(active, starting, since 0.00300597s) 2023-10-09T07:39:15.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:15 smithi115 ceph-mon[108015]: from='mgr.14100 172.21.15.115:0/3616151999' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-10-09T07:39:15.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:15 smithi115 ceph-mon[108015]: from='mgr.14100 172.21.15.115:0/3616151999' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-10-09T07:39:15.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:15 smithi115 ceph-mon[108015]: from='mgr.14100 172.21.15.115:0/3616151999' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-10-09T07:39:15.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:15 smithi115 ceph-mon[108015]: from='mgr.14100 172.21.15.115:0/3616151999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-09T07:39:15.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:15 smithi115 ceph-mon[108015]: from='mgr.14100 172.21.15.115:0/3616151999' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-10-09T07:39:15.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:15 smithi115 ceph-mon[108015]: Manager daemon x is now available 2023-10-09T07:39:15.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:15 smithi115 ceph-mon[108015]: from='mgr.14100 172.21.15.115:0/3616151999' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-10-09T07:39:15.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:15 smithi115 ceph-mon[108015]: from='mgr.14100 172.21.15.115:0/3616151999' entity='mgr.x' 2023-10-09T07:39:15.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:15 smithi115 ceph-mon[108015]: from='mgr.14100 172.21.15.115:0/3616151999' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-10-09T07:39:15.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:15 smithi115 ceph-mon[108015]: from='mgr.14100 172.21.15.115:0/3616151999' entity='mgr.x' 2023-10-09T07:39:15.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:15 smithi115 ceph-mon[108015]: from='mgr.14100 172.21.15.115:0/3616151999' entity='mgr.x' 2023-10-09T07:39:15.978 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-10-09T07:39:15.978 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout { 2023-10-09T07:39:15.978 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "fsid": "d56ccb5c-6676-11ee-8db5-212e2dc638e7", 2023-10-09T07:39:15.978 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "health": { 2023-10-09T07:39:15.978 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-10-09T07:39:15.978 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-10-09T07:39:15.978 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-10-09T07:39:15.978 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:15.979 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-10-09T07:39:15.979 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-10-09T07:39:15.979 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 0 2023-10-09T07:39:15.979 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout ], 2023-10-09T07:39:15.979 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-10-09T07:39:15.979 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "a" 2023-10-09T07:39:15.979 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout ], 2023-10-09T07:39:15.979 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2023-10-09T07:39:15.979 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "monmap": { 2023-10-09T07:39:15.979 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-09T07:39:15.979 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-10-09T07:39:15.979 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-10-09T07:39:15.980 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:15.980 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-10-09T07:39:15.980 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-09T07:39:15.980 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-10-09T07:39:15.980 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-10-09T07:39:15.980 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-10-09T07:39:15.980 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-10-09T07:39:15.983 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-10-09T07:39:15.983 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-10-09T07:39:15.983 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:15.983 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-10-09T07:39:15.983 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-10-09T07:39:15.984 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-10-09T07:39:15.984 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-10-09T07:39:15.984 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-10-09T07:39:15.984 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-10-09T07:39:15.984 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-10-09T07:39:15.984 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-10-09T07:39:15.984 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-10-09T07:39:15.984 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:15.984 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-10-09T07:39:15.984 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-09T07:39:15.984 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-10-09T07:39:15.985 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-10-09T07:39:15.985 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:15.985 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-10-09T07:39:15.985 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "available": false, 2023-10-09T07:39:15.985 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-10-09T07:39:15.985 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "modules": [ 2023-10-09T07:39:15.985 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "iostat", 2023-10-09T07:39:15.985 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "nfs", 2023-10-09T07:39:15.985 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "restful" 2023-10-09T07:39:15.985 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout ], 2023-10-09T07:39:15.986 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-09T07:39:15.986 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:15.986 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-10-09T07:39:15.986 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-09T07:39:15.986 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "modified": "2023-10-09T07:39:04.104242+0000", 2023-10-09T07:39:15.986 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-09T07:39:15.987 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:15.987 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-10-09T07:39:15.987 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout } 2023-10-09T07:39:15.987 INFO:teuthology.orchestra.run.smithi115.stdout:mgr not available, waiting (3/15)... 2023-10-09T07:39:16.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:16 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/2950522506' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-10-09T07:39:16.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:16 smithi115 ceph-mon[108015]: mgrmap e3: x(active, since 1.00582s) 2023-10-09T07:39:18.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:18 smithi115 ceph-mon[108015]: mgrmap e4: x(active, since 2s) 2023-10-09T07:39:18.884 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-10-09T07:39:18.884 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout { 2023-10-09T07:39:18.884 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "fsid": "d56ccb5c-6676-11ee-8db5-212e2dc638e7", 2023-10-09T07:39:18.884 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "health": { 2023-10-09T07:39:18.884 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-10-09T07:39:18.884 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-10-09T07:39:18.884 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-10-09T07:39:18.884 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:18.885 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-10-09T07:39:18.885 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-10-09T07:39:18.885 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 0 2023-10-09T07:39:18.885 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout ], 2023-10-09T07:39:18.885 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-10-09T07:39:18.885 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "a" 2023-10-09T07:39:18.885 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout ], 2023-10-09T07:39:18.885 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2023-10-09T07:39:18.885 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "monmap": { 2023-10-09T07:39:18.885 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-09T07:39:18.885 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-10-09T07:39:18.886 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-10-09T07:39:18.886 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:18.886 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-10-09T07:39:18.886 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-09T07:39:18.886 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-10-09T07:39:18.886 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-10-09T07:39:18.886 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-10-09T07:39:18.886 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-10-09T07:39:18.886 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-10-09T07:39:18.886 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-10-09T07:39:18.886 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:18.886 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-10-09T07:39:18.887 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-10-09T07:39:18.887 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-10-09T07:39:18.887 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-10-09T07:39:18.887 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-10-09T07:39:18.887 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-10-09T07:39:18.887 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-10-09T07:39:18.887 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-10-09T07:39:18.887 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-10-09T07:39:18.887 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:18.887 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-10-09T07:39:18.887 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-09T07:39:18.887 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-10-09T07:39:18.887 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-10-09T07:39:18.888 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:18.888 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-10-09T07:39:18.888 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "available": true, 2023-10-09T07:39:18.888 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-10-09T07:39:18.888 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "modules": [ 2023-10-09T07:39:18.888 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "iostat", 2023-10-09T07:39:18.888 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "nfs", 2023-10-09T07:39:18.888 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "restful" 2023-10-09T07:39:18.888 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout ], 2023-10-09T07:39:18.888 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-09T07:39:18.888 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:18.888 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-10-09T07:39:18.889 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-09T07:39:18.889 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "modified": "2023-10-09T07:39:04.104242+0000", 2023-10-09T07:39:18.889 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-09T07:39:18.889 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-10-09T07:39:18.889 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-10-09T07:39:18.889 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout } 2023-10-09T07:39:18.889 INFO:teuthology.orchestra.run.smithi115.stdout:mgr is available 2023-10-09T07:39:19.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:19 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/2985522949' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-10-09T07:39:19.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:19 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/2032735495' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-10-09T07:39:19.659 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-10-09T07:39:19.659 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout [global] 2023-10-09T07:39:19.659 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout fsid = d56ccb5c-6676-11ee-8db5-212e2dc638e7 2023-10-09T07:39:19.659 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-10-09T07:39:19.659 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-10-09T07:39:19.659 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-10-09T07:39:19.660 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-10-09T07:39:19.660 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-10-09T07:39:19.660 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout [mgr] 2023-10-09T07:39:19.660 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-10-09T07:39:19.660 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-10-09T07:39:19.660 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout [osd] 2023-10-09T07:39:19.660 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-10-09T07:39:19.660 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-10-09T07:39:19.660 INFO:teuthology.orchestra.run.smithi115.stdout:Enabling cephadm module... 2023-10-09T07:39:20.778 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:20 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/1297106807' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-10-09T07:39:20.778 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:20 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: ignoring --setuser ceph since I am not root 2023-10-09T07:39:20.778 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:20 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: ignoring --setgroup ceph since I am not root 2023-10-09T07:39:20.778 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:20 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:20.654+0000 7f543eee9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-09T07:39:21.102 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:20 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:20.823+0000 7f543eee9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-09T07:39:21.486 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:21 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:21.168+0000 7f543eee9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-09T07:39:21.683 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout { 2023-10-09T07:39:21.683 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-10-09T07:39:21.683 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "available": true, 2023-10-09T07:39:21.683 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-10-09T07:39:21.684 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-10-09T07:39:21.684 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout } 2023-10-09T07:39:21.684 INFO:teuthology.orchestra.run.smithi115.stdout:Waiting for the mgr to restart... 2023-10-09T07:39:21.684 INFO:teuthology.orchestra.run.smithi115.stdout:Waiting for mgr epoch 5... 2023-10-09T07:39:21.770 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:21 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:21.676+0000 7f543eee9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-09T07:39:21.770 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:21 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/1297106807' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-10-09T07:39:21.770 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:21 smithi115 ceph-mon[108015]: mgrmap e5: x(active, since 5s) 2023-10-09T07:39:21.770 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:21 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/2666092988' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-10-09T07:39:22.102 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:21 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:21.913+0000 7f543eee9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-09T07:39:22.602 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:22 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:22.335+0000 7f543eee9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-09T07:39:23.352 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:23 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:23.030+0000 7f543eee9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-09T07:39:23.352 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:23 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:23.117+0000 7f543eee9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-09T07:39:24.102 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:23 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:23.708+0000 7f543eee9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-09T07:39:24.102 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:23 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:23.793+0000 7f543eee9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-09T07:39:24.102 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:23 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:23.880+0000 7f543eee9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-09T07:39:24.510 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:24 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:24.110+0000 7f543eee9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-09T07:39:24.511 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:24 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:24.203+0000 7f543eee9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-09T07:39:24.511 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:24 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:24.295+0000 7f543eee9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-09T07:39:24.511 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:24 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:24.509+0000 7f543eee9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-09T07:39:24.789 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:24 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:24.605+0000 7f543eee9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-09T07:39:25.102 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:24 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:24.788+0000 7f543eee9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-09T07:39:25.102 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:24 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:24.874+0000 7f543eee9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-09T07:39:25.997 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:25 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:25.621+0000 7f543eee9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-09T07:39:25.997 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:25 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:25.816+0000 7f543eee9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-09T07:39:25.997 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:25 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:25.996+0000 7f543eee9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-09T07:39:27.009 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:26 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:26.647+0000 7f543eee9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-09T07:39:27.009 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:26 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:26.828+0000 7f543eee9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-09T07:39:27.352 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:27 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:27.008+0000 7f543eee9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-09T07:39:27.352 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:27 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:27.124+0000 7f543eee9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-09T07:39:27.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:27 smithi115 ceph-mon[108015]: Active manager daemon x restarted 2023-10-09T07:39:27.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:27 smithi115 ceph-mon[108015]: Activating manager daemon x 2023-10-09T07:39:27.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:27 smithi115 ceph-mon[108015]: osdmap e2: 0 total, 0 up, 0 in 2023-10-09T07:39:27.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:27 smithi115 ceph-mon[108015]: mgrmap e6: x(active, starting, since 0.0031216s) 2023-10-09T07:39:27.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:27 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-09T07:39:27.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:27 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-10-09T07:39:27.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:27 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-10-09T07:39:27.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:27 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-10-09T07:39:27.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:27 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-10-09T07:39:27.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:27 smithi115 ceph-mon[108015]: Manager daemon x is now available 2023-10-09T07:39:27.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:27 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' 2023-10-09T07:39:27.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:27 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' 2023-10-09T07:39:27.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:27 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:39:27.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:27 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:39:28.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:28 smithi115 ceph-mon[108015]: Found migration_current of "None". Setting to last migration. 2023-10-09T07:39:28.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:28 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:39:28.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:28 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-10-09T07:39:28.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:28 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-10-09T07:39:28.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:28 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' 2023-10-09T07:39:28.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:28 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' 2023-10-09T07:39:28.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:28 smithi115 ceph-mon[108015]: mgrmap e7: x(active, since 1.00734s) 2023-10-09T07:39:29.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:29 smithi115 ceph-mon[108015]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-10-09T07:39:29.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:29 smithi115 ceph-mon[108015]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-10-09T07:39:29.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:29 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:39:30.381 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout { 2023-10-09T07:39:30.382 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-10-09T07:39:30.382 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "initialized": true 2023-10-09T07:39:30.382 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout } 2023-10-09T07:39:30.382 INFO:teuthology.orchestra.run.smithi115.stdout:mgr epoch 5 is available 2023-10-09T07:39:30.382 INFO:teuthology.orchestra.run.smithi115.stdout:Setting orchestrator backend to cephadm... 2023-10-09T07:39:30.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:30 smithi115 ceph-mon[108015]: [09/Oct/2023:07:39:28] ENGINE Bus STARTING 2023-10-09T07:39:30.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:30 smithi115 ceph-mon[108015]: [09/Oct/2023:07:39:28] ENGINE Serving on http://172.21.15.115:8765 2023-10-09T07:39:30.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:30 smithi115 ceph-mon[108015]: [09/Oct/2023:07:39:29] ENGINE Serving on https://172.21.15.115:7150 2023-10-09T07:39:30.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:30 smithi115 ceph-mon[108015]: [09/Oct/2023:07:39:29] ENGINE Bus STARTED 2023-10-09T07:39:30.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:30 smithi115 ceph-mon[108015]: mgrmap e8: x(active, since 2s) 2023-10-09T07:39:33.033 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout value unchanged 2023-10-09T07:39:33.033 INFO:teuthology.orchestra.run.smithi115.stdout:Generating ssh key... 2023-10-09T07:39:33.034 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:32 smithi115 ceph-mon[108015]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-10-09T07:39:33.034 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:32 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' 2023-10-09T07:39:33.034 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:32 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:39:34.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:33 smithi115 ceph-mon[108015]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-10-09T07:39:34.157 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:33 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: Generating public/private rsa key pair. 2023-10-09T07:39:34.157 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:33 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: Your identification has been saved in /tmp/tmp8_363vmv/key. 2023-10-09T07:39:34.157 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:33 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: Your public key has been saved in /tmp/tmp8_363vmv/key.pub. 2023-10-09T07:39:34.157 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:33 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: The key fingerprint is: 2023-10-09T07:39:34.157 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:33 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: SHA256:7ood0UqSwv/aX9JHNGa5rAtIhXIN9w/e/JQCKm0Y83o ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7 2023-10-09T07:39:34.157 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:33 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: The key's randomart image is: 2023-10-09T07:39:34.157 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:33 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: +---[RSA 3072]----+ 2023-10-09T07:39:34.158 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:33 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: | . . | 2023-10-09T07:39:34.158 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:33 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: | = . . | 2023-10-09T07:39:34.158 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:33 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: | . = o + * | 2023-10-09T07:39:34.158 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:33 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: | . + O o @ o . | 2023-10-09T07:39:34.158 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:33 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: | o o * S . O o | 2023-10-09T07:39:34.158 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:33 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: | o + O . o + | 2023-10-09T07:39:34.158 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:33 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: | . = E + . . | 2023-10-09T07:39:34.158 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:33 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: | = + + o | 2023-10-09T07:39:34.158 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:33 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: | o.=oo . | 2023-10-09T07:39:34.159 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:33 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: +----[SHA256]-----+ 2023-10-09T07:39:35.034 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:34 smithi115 ceph-mon[108015]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-10-09T07:39:35.034 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:34 smithi115 ceph-mon[108015]: Generating ssh key... 2023-10-09T07:39:35.035 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:34 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' 2023-10-09T07:39:35.035 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:34 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' 2023-10-09T07:39:35.071 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDPe9JRkXhyK3W9gFTyscuYtVcU6AjEck9al0wOhLd3Ibp2EkGhhl25dRKq8gDVhWZmcivTJ/7VpKNWpEFeSh52ZyL7o5ncvPQh7p8XihBwaHawkufi078uC4Juw0+MJq4OZTvybc/idpJTQzpnqIX5P1NCMQqjSU2z3srGQR/7eEkgLBroXPHsnO3GyWxklCANCQ9OHnpixXK0Mp757Gc21YeaiO596oQt4xDRP5CjCk9zEY65nLypS7eK4V1wBVSdJF30BFCrVWV76iXuYtqzaQ5sxkGI1QLTMneOOO8tsjmHdg67nhVrnqQVJSl0z0AsZFwKkQ5g5Z23GsgUYdl3EQs+CpdSlehsKV3FdWWJTrQepNG/R2MYSEWBSfuImyfaR4nC3dEfda7xLkvZr6w9egODBSyZW5DLrn+MTXc5tht6BFeJHC6KZMdASELNKCgtf6VvrKDeAQocpb4fbOXwhUthukjYTlQg0c/fKKRj+RPdwekRdDx+Crz9b76IJH8= ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7 2023-10-09T07:39:35.071 INFO:teuthology.orchestra.run.smithi115.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-10-09T07:39:35.072 INFO:teuthology.orchestra.run.smithi115.stdout:Adding key to root@localhost authorized_keys... 2023-10-09T07:39:35.072 INFO:teuthology.orchestra.run.smithi115.stdout:Adding host smithi115... 2023-10-09T07:39:36.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:35 smithi115 ceph-mon[108015]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-10-09T07:39:36.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:35 smithi115 ceph-mon[108015]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi115", "addr": "172.21.15.115", "target": ["mon-mgr", ""]}]: dispatch 2023-10-09T07:39:37.225 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:36 smithi115 ceph-mon[108015]: Deploying cephadm binary to smithi115 2023-10-09T07:39:37.986 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout Added host 'smithi115' with addr '172.21.15.115' 2023-10-09T07:39:37.986 INFO:teuthology.orchestra.run.smithi115.stdout:Deploying unmanaged mon service... 2023-10-09T07:39:38.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:38 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' 2023-10-09T07:39:38.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:38 smithi115 ceph-mon[108015]: Added host smithi115 2023-10-09T07:39:38.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:38 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:39:38.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:38 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' 2023-10-09T07:39:39.037 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-10-09T07:39:39.037 INFO:teuthology.orchestra.run.smithi115.stdout:Deploying unmanaged mgr service... 2023-10-09T07:39:40.061 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-10-09T07:39:40.225 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:40 smithi115 ceph-mon[108015]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-09T07:39:40.225 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:40 smithi115 ceph-mon[108015]: Saving service mon spec with placement count:5 2023-10-09T07:39:40.226 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:40 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' 2023-10-09T07:39:40.226 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:40 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' 2023-10-09T07:39:40.226 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:40 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' 2023-10-09T07:39:41.730 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:41 smithi115 ceph-mon[108015]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-09T07:39:41.730 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:41 smithi115 ceph-mon[108015]: Saving service mgr spec with placement count:2 2023-10-09T07:39:41.731 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:41 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' 2023-10-09T07:39:41.731 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:41 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/1185063657' entity='client.admin' 2023-10-09T07:39:41.731 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:41 smithi115 ceph-mon[108015]: from='mgr.14120 172.21.15.115:0/3670509352' entity='mgr.x' 2023-10-09T07:39:41.871 INFO:teuthology.orchestra.run.smithi115.stdout:Enabling the dashboard module... 2023-10-09T07:39:42.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:42 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/2456578809' entity='client.admin' 2023-10-09T07:39:42.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:42 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/3482317671' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-10-09T07:39:43.797 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: ignoring --setuser ceph since I am not root 2023-10-09T07:39:43.797 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: ignoring --setgroup ceph since I am not root 2023-10-09T07:39:43.798 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:43.627+0000 7fae240c1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-09T07:39:43.798 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:43.797+0000 7fae240c1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-09T07:39:44.441 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:44 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:44.142+0000 7fae240c1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-09T07:39:44.669 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout { 2023-10-09T07:39:44.670 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-10-09T07:39:44.670 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "available": true, 2023-10-09T07:39:44.670 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-10-09T07:39:44.670 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-10-09T07:39:44.670 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout } 2023-10-09T07:39:44.670 INFO:teuthology.orchestra.run.smithi115.stdout:Waiting for the mgr to restart... 2023-10-09T07:39:44.670 INFO:teuthology.orchestra.run.smithi115.stdout:Waiting for mgr epoch 9... 2023-10-09T07:39:44.759 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:44 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:44.648+0000 7fae240c1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-09T07:39:44.759 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:44 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/3482317671' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-10-09T07:39:44.759 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:44 smithi115 ceph-mon[108015]: mgrmap e9: x(active, since 16s) 2023-10-09T07:39:44.759 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:44 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/1628653909' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-10-09T07:39:45.102 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:44 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:44.890+0000 7fae240c1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-09T07:39:45.602 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:45 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:45.310+0000 7fae240c1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-09T07:39:46.352 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:46 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:46.017+0000 7fae240c1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-09T07:39:46.352 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:46 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:46.104+0000 7fae240c1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-09T07:39:47.102 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:46 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:46.702+0000 7fae240c1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-09T07:39:47.102 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:46 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:46.787+0000 7fae240c1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-09T07:39:47.102 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:46 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:46.876+0000 7fae240c1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-09T07:39:47.517 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:47 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:47.115+0000 7fae240c1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-09T07:39:47.517 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:47 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:47.207+0000 7fae240c1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-09T07:39:47.517 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:47 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:47.301+0000 7fae240c1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-09T07:39:47.517 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:47 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:47.516+0000 7fae240c1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-09T07:39:47.797 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:47 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:47.613+0000 7fae240c1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-09T07:39:48.102 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:47 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:47.797+0000 7fae240c1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-09T07:39:48.102 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:47 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:47.882+0000 7fae240c1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-09T07:39:48.999 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:48 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:48.626+0000 7fae240c1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-09T07:39:48.999 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:48 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:48.822+0000 7fae240c1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-09T07:39:50.014 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:48 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:48.998+0000 7fae240c1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-09T07:39:50.014 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:49 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:49.649+0000 7fae240c1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-09T07:39:50.014 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:49 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:49.830+0000 7fae240c1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-09T07:39:50.352 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:50 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:50.014+0000 7fae240c1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-09T07:39:50.352 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:39:50 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:39:50.129+0000 7fae240c1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-09T07:39:50.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:50 smithi115 ceph-mon[108015]: Active manager daemon x restarted 2023-10-09T07:39:50.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:50 smithi115 ceph-mon[108015]: Activating manager daemon x 2023-10-09T07:39:50.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:50 smithi115 ceph-mon[108015]: osdmap e3: 0 total, 0 up, 0 in 2023-10-09T07:39:50.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:50 smithi115 ceph-mon[108015]: mgrmap e10: x(active, starting, since 0.00317949s) 2023-10-09T07:39:50.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:50 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-09T07:39:50.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:50 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-10-09T07:39:50.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:50 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-10-09T07:39:50.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:50 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-10-09T07:39:50.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:50 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-10-09T07:39:50.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:50 smithi115 ceph-mon[108015]: Manager daemon x is now available 2023-10-09T07:39:50.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:50 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:39:50.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:50 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:39:51.445 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:51 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-10-09T07:39:51.445 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:51 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-10-09T07:39:51.445 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:51 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:39:51.445 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:51 smithi115 ceph-mon[108015]: mgrmap e11: x(active, since 1.00596s) 2023-10-09T07:39:51.544 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout { 2023-10-09T07:39:51.544 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-10-09T07:39:51.544 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "initialized": true 2023-10-09T07:39:51.544 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout } 2023-10-09T07:39:51.544 INFO:teuthology.orchestra.run.smithi115.stdout:mgr epoch 9 is available 2023-10-09T07:39:51.544 INFO:teuthology.orchestra.run.smithi115.stdout:Generating a dashboard self-signed certificate... 2023-10-09T07:39:52.258 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:52 smithi115 ceph-mon[108015]: [09/Oct/2023:07:39:50] ENGINE Bus STARTING 2023-10-09T07:39:52.258 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:52 smithi115 ceph-mon[108015]: [09/Oct/2023:07:39:50] ENGINE Serving on http://172.21.15.115:8765 2023-10-09T07:39:52.258 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:52 smithi115 ceph-mon[108015]: [09/Oct/2023:07:39:50] ENGINE Serving on https://172.21.15.115:7150 2023-10-09T07:39:52.258 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:52 smithi115 ceph-mon[108015]: [09/Oct/2023:07:39:50] ENGINE Bus STARTED 2023-10-09T07:39:52.258 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:52 smithi115 ceph-mon[108015]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-10-09T07:39:52.258 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:52 smithi115 ceph-mon[108015]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-10-09T07:39:52.258 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:52 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:39:52.767 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-10-09T07:39:52.767 INFO:teuthology.orchestra.run.smithi115.stdout:Creating initial admin user... 2023-10-09T07:39:53.483 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:53 smithi115 ceph-mon[108015]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-10-09T07:39:53.483 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:53 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:39:53.483 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:53 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:39:53.483 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:53 smithi115 ceph-mon[108015]: mgrmap e12: x(active, since 2s) 2023-10-09T07:39:53.801 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$3L3mBwW4rzUySSkckg5j3e6zm3PlmHSXUBuPVf6oraqUI2Lx3rIQi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1696837193, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-10-09T07:39:53.801 INFO:teuthology.orchestra.run.smithi115.stdout:Fetching dashboard port number... 2023-10-09T07:39:54.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:54 smithi115 ceph-mon[108015]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-09T07:39:54.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:54 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:39:54.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:54 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/1993422352' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-10-09T07:39:54.616 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 8443 2023-10-09T07:39:54.625 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-10-09T07:39:54.625 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout disabled 2023-10-09T07:39:54.633 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-10-09T07:39:54.633 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout inactive 2023-10-09T07:39:54.633 INFO:teuthology.orchestra.run.smithi115.stdout:firewalld.service is not enabled 2023-10-09T07:39:54.634 INFO:teuthology.orchestra.run.smithi115.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-10-09T07:39:54.635 INFO:teuthology.orchestra.run.smithi115.stdout:Ceph Dashboard is now available at: 2023-10-09T07:39:54.635 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:39:54.635 INFO:teuthology.orchestra.run.smithi115.stdout: URL: https://smithi115.front.sepia.ceph.com:8443/ 2023-10-09T07:39:54.635 INFO:teuthology.orchestra.run.smithi115.stdout: User: admin 2023-10-09T07:39:54.635 INFO:teuthology.orchestra.run.smithi115.stdout: Password: eg3gyat1ff 2023-10-09T07:39:54.635 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:39:54.636 INFO:teuthology.orchestra.run.smithi115.stdout:Saving cluster configuration to /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/config directory 2023-10-09T07:39:54.636 INFO:teuthology.orchestra.run.smithi115.stdout:Enabling autotune for osd_memory_target 2023-10-09T07:39:56.467 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-10-09T07:39:56.467 INFO:teuthology.orchestra.run.smithi115.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-10-09T07:39:56.467 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:39:56.467 INFO:teuthology.orchestra.run.smithi115.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-10-09T07:39:56.467 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:39:56.467 INFO:teuthology.orchestra.run.smithi115.stdout:Or, if you are only running a single cluster on this host: 2023-10-09T07:39:56.467 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:39:56.468 INFO:teuthology.orchestra.run.smithi115.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-10-09T07:39:56.468 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:39:56.468 INFO:teuthology.orchestra.run.smithi115.stdout:Please consider enabling telemetry to help improve Ceph: 2023-10-09T07:39:56.468 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:39:56.468 INFO:teuthology.orchestra.run.smithi115.stdout: ceph telemetry on 2023-10-09T07:39:56.468 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:39:56.468 INFO:teuthology.orchestra.run.smithi115.stdout:For more information see: 2023-10-09T07:39:56.468 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:39:56.468 INFO:teuthology.orchestra.run.smithi115.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-10-09T07:39:56.469 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:39:56.469 INFO:teuthology.orchestra.run.smithi115.stdout:Bootstrap complete. 2023-10-09T07:39:56.501 INFO:tasks.cephadm:Fetching config... 2023-10-09T07:39:56.501 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-10-09T07:39:56.502 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-10-09T07:39:56.520 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-10-09T07:39:56.520 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-10-09T07:39:56.520 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-10-09T07:39:56.579 INFO:tasks.cephadm:Fetching mon keyring... 2023-10-09T07:39:56.579 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-10-09T07:39:56.579 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/keyring of=/dev/stdout 2023-10-09T07:39:56.648 INFO:tasks.cephadm:Fetching pub ssh key... 2023-10-09T07:39:56.648 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-10-09T07:39:56.648 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-10-09T07:39:56.706 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-10-09T07:39:56.706 DEBUG:teuthology.orchestra.run.smithi115:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDPe9JRkXhyK3W9gFTyscuYtVcU6AjEck9al0wOhLd3Ibp2EkGhhl25dRKq8gDVhWZmcivTJ/7VpKNWpEFeSh52ZyL7o5ncvPQh7p8XihBwaHawkufi078uC4Juw0+MJq4OZTvybc/idpJTQzpnqIX5P1NCMQqjSU2z3srGQR/7eEkgLBroXPHsnO3GyWxklCANCQ9OHnpixXK0Mp757Gc21YeaiO596oQt4xDRP5CjCk9zEY65nLypS7eK4V1wBVSdJF30BFCrVWV76iXuYtqzaQ5sxkGI1QLTMneOOO8tsjmHdg67nhVrnqQVJSl0z0AsZFwKkQ5g5Z23GsgUYdl3EQs+CpdSlehsKV3FdWWJTrQepNG/R2MYSEWBSfuImyfaR4nC3dEfda7xLkvZr6w9egODBSyZW5DLrn+MTXc5tht6BFeJHC6KZMdASELNKCgtf6VvrKDeAQocpb4fbOXwhUthukjYTlQg0c/fKKRj+RPdwekRdDx+Crz9b76IJH8= ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-10-09T07:39:56.785 INFO:teuthology.orchestra.run.smithi115.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDPe9JRkXhyK3W9gFTyscuYtVcU6AjEck9al0wOhLd3Ibp2EkGhhl25dRKq8gDVhWZmcivTJ/7VpKNWpEFeSh52ZyL7o5ncvPQh7p8XihBwaHawkufi078uC4Juw0+MJq4OZTvybc/idpJTQzpnqIX5P1NCMQqjSU2z3srGQR/7eEkgLBroXPHsnO3GyWxklCANCQ9OHnpixXK0Mp757Gc21YeaiO596oQt4xDRP5CjCk9zEY65nLypS7eK4V1wBVSdJF30BFCrVWV76iXuYtqzaQ5sxkGI1QLTMneOOO8tsjmHdg67nhVrnqQVJSl0z0AsZFwKkQ5g5Z23GsgUYdl3EQs+CpdSlehsKV3FdWWJTrQepNG/R2MYSEWBSfuImyfaR4nC3dEfda7xLkvZr6w9egODBSyZW5DLrn+MTXc5tht6BFeJHC6KZMdASELNKCgtf6VvrKDeAQocpb4fbOXwhUthukjYTlQg0c/fKKRj+RPdwekRdDx+Crz9b76IJH8= ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7 2023-10-09T07:39:56.796 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-10-09T07:39:57.053 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:39:57.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:57 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/2901702896' entity='client.admin' 2023-10-09T07:39:58.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:58 smithi115 ceph-mon[108015]: mgrmap e13: x(active, since 6s) 2023-10-09T07:39:58.694 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-10-09T07:39:58.694 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-10-09T07:39:59.119 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:39:59.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:59 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/55343189' entity='client.admin' 2023-10-09T07:39:59.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:59 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:39:59.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:59 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:39:59.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:39:59 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi115", "name": "osd_memory_target"}]: dispatch 2023-10-09T07:40:00.568 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:00 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:00.568 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:00 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:40:00.568 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:00 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:00.568 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:00 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:00.568 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:00 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:40:00.568 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:00 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:40:00.569 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:00 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:40:00.606 INFO:tasks.cephadm:Setting crush tunables to default 2023-10-09T07:40:00.606 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph osd crush tunables default 2023-10-09T07:40:00.839 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:40:01.581 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:01 smithi115 ceph-mon[108015]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-10-09T07:40:02.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:01 smithi115 ceph-mon[108015]: Updating smithi115:/etc/ceph/ceph.conf 2023-10-09T07:40:02.360 INFO:teuthology.orchestra.run.smithi115.stderr:adjusted tunables profile to default 2023-10-09T07:40:02.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:02 smithi115 ceph-mon[108015]: Updating smithi115:/var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/config/ceph.conf 2023-10-09T07:40:02.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:02 smithi115 ceph-mon[108015]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2023-10-09T07:40:02.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:02 smithi115 ceph-mon[108015]: Updating smithi115:/var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-09T07:40:02.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:02 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:02.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:02 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:02.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:02 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:02.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:02 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/3892780696' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-10-09T07:40:03.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:03 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/3892780696' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-10-09T07:40:03.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:03 smithi115 ceph-mon[108015]: osdmap e4: 0 total, 0 up, 0 in 2023-10-09T07:40:04.175 INFO:tasks.cephadm:Adding mon.a on smithi115 2023-10-09T07:40:04.175 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch apply mon '1;smithi115:172.21.15.115=a' 2023-10-09T07:40:04.394 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:40:05.411 INFO:teuthology.orchestra.run.smithi115.stdout:Scheduled mon update... 2023-10-09T07:40:05.859 INFO:tasks.cephadm:Waiting for 1 mons in monmap... 2023-10-09T07:40:05.859 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph mon dump -f json 2023-10-09T07:40:06.315 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:40:06.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:06 smithi115 ceph-mon[108015]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "1;smithi115:172.21.15.115=a", "target": ["mon-mgr", ""]}]: dispatch 2023-10-09T07:40:06.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:06 smithi115 ceph-mon[108015]: Saving service mon spec with placement smithi115:172.21.15.115=a;count:1 2023-10-09T07:40:06.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:06 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:06.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:06 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:40:06.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:06 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:40:06.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:06 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:40:06.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:06 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:06.546 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:06 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:06.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:06 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:06.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:06 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:06.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:06 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:06.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:06 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:06.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:06 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-09T07:40:06.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:06 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-09T07:40:06.547 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:06 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:40:07.525 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:40:07.525 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"d56ccb5c-6676-11ee-8db5-212e2dc638e7","modified":"2023-10-09T07:39:02.290778Z","created":"2023-10-09T07:39:02.290778Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:3300","nonce":0},{"type":"v1","addr":"172.21.15.115:6789","nonce":0}]},"addr":"172.21.15.115:6789/0","public_addr":"172.21.15.115:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-09T07:40:07.527 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2023-10-09T07:40:07.816 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:07 smithi115 ceph-mon[108015]: Reconfiguring mon.a (unknown last config time)... 2023-10-09T07:40:07.817 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:07 smithi115 ceph-mon[108015]: Reconfiguring daemon mon.a on smithi115 2023-10-09T07:40:07.817 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:07 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:07.817 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:07 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:07.893 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-10-09T07:40:07.893 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph config generate-minimal-conf 2023-10-09T07:40:08.107 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:40:08.487 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:08 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/4190459189' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-09T07:40:09.152 INFO:teuthology.orchestra.run.smithi115.stdout:# minimal ceph.conf for d56ccb5c-6676-11ee-8db5-212e2dc638e7 2023-10-09T07:40:09.153 INFO:teuthology.orchestra.run.smithi115.stdout:[global] 2023-10-09T07:40:09.153 INFO:teuthology.orchestra.run.smithi115.stdout: fsid = d56ccb5c-6676-11ee-8db5-212e2dc638e7 2023-10-09T07:40:09.153 INFO:teuthology.orchestra.run.smithi115.stdout: mon_host = [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2023-10-09T07:40:09.601 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-10-09T07:40:09.601 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-10-09T07:40:09.601 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.conf 2023-10-09T07:40:09.630 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-10-09T07:40:09.630 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-10-09T07:40:09.697 INFO:tasks.cephadm:Adding mgr.x on smithi115 2023-10-09T07:40:09.697 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch apply mgr '1;smithi115=x' 2023-10-09T07:40:09.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:09 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/3696657532' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:40:09.947 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:40:10.992 INFO:teuthology.orchestra.run.smithi115.stdout:Scheduled mgr update... 2023-10-09T07:40:11.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:11 smithi115 ceph-mon[108015]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-09T07:40:11.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:11 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:11.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:11 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:40:11.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:11 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:40:11.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:11 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:40:11.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:11 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:11.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:11 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:11.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:11 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-10-09T07:40:11.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:11 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-10-09T07:40:11.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:11 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:40:11.354 INFO:tasks.cephadm:Deploying OSDs... 2023-10-09T07:40:11.354 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-10-09T07:40:11.354 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/scratch_devs of=/dev/stdout 2023-10-09T07:40:11.413 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-09T07:40:11.413 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_1 2023-10-09T07:40:11.471 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-09T07:40:11.471 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-09T07:40:11.471 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 390104 Links: 1 2023-10-09T07:40:11.471 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-09T07:40:11.471 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2023-10-09T07:40:11.471 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2023-10-09 07:39:58.663200839 +0000 2023-10-09T07:40:11.471 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2023-10-09 07:37:34.989043791 +0000 2023-10-09T07:40:11.471 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2023-10-09 07:37:34.989043791 +0000 2023-10-09T07:40:11.471 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2023-10-09T07:40:11.472 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-09T07:40:11.539 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2023-10-09T07:40:11.540 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2023-10-09T07:40:11.540 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000206017 s, 2.5 MB/s 2023-10-09T07:40:11.541 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-09T07:40:11.600 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_2 2023-10-09T07:40:11.658 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-09T07:40:11.658 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-09T07:40:11.659 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 390144 Links: 1 2023-10-09T07:40:11.659 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-09T07:40:11.659 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2023-10-09T07:40:11.659 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2023-10-09 07:39:58.663200839 +0000 2023-10-09T07:40:11.659 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2023-10-09 07:37:35.524033277 +0000 2023-10-09T07:40:11.659 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2023-10-09 07:37:35.524033277 +0000 2023-10-09T07:40:11.659 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2023-10-09T07:40:11.659 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-09T07:40:11.725 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2023-10-09T07:40:11.726 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2023-10-09T07:40:11.726 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000212722 s, 2.4 MB/s 2023-10-09T07:40:11.727 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-09T07:40:11.786 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_3 2023-10-09T07:40:11.844 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-09T07:40:11.844 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-09T07:40:11.844 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 393723 Links: 1 2023-10-09T07:40:11.844 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-09T07:40:11.845 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2023-10-09T07:40:11.845 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2023-10-09 07:39:58.663200839 +0000 2023-10-09T07:40:11.845 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2023-10-09 07:37:36.017023589 +0000 2023-10-09T07:40:11.845 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2023-10-09 07:37:36.017023589 +0000 2023-10-09T07:40:11.845 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2023-10-09T07:40:11.845 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-09T07:40:11.912 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2023-10-09T07:40:11.912 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2023-10-09T07:40:11.912 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000208053 s, 2.5 MB/s 2023-10-09T07:40:11.913 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-09T07:40:11.973 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_4 2023-10-09T07:40:12.031 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-09T07:40:12.031 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-09T07:40:12.031 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 396850 Links: 1 2023-10-09T07:40:12.031 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-09T07:40:12.031 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2023-10-09T07:40:12.032 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2023-10-09 07:39:58.663200839 +0000 2023-10-09T07:40:12.032 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2023-10-09 07:37:36.539013331 +0000 2023-10-09T07:40:12.032 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2023-10-09 07:37:36.539013331 +0000 2023-10-09T07:40:12.032 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2023-10-09T07:40:12.032 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-09T07:40:12.097 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2023-10-09T07:40:12.097 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2023-10-09T07:40:12.097 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000281667 s, 1.8 MB/s 2023-10-09T07:40:12.098 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-09T07:40:12.157 INFO:tasks.cephadm:Deploying osd.0 on smithi115 with /dev/vg_nvme/lv_4... 2023-10-09T07:40:12.157 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-10-09T07:40:12.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:12 smithi115 ceph-mon[108015]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "1;smithi115=x", "target": ["mon-mgr", ""]}]: dispatch 2023-10-09T07:40:12.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:12 smithi115 ceph-mon[108015]: Saving service mgr spec with placement smithi115=x;count:1 2023-10-09T07:40:12.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:12 smithi115 ceph-mon[108015]: Reconfiguring mgr.x (unknown last config time)... 2023-10-09T07:40:12.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:12 smithi115 ceph-mon[108015]: Reconfiguring daemon mgr.x on smithi115 2023-10-09T07:40:12.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:12 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:12.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:12 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:12.408 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:40:13.544 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-10-09T07:40:13.544 INFO:teuthology.orchestra.run.smithi115.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-10-09T07:40:13.545 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10+0 records in 2023-10-09T07:40:13.545 INFO:teuthology.orchestra.run.smithi115.stderr:10+0 records out 2023-10-09T07:40:13.545 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112323 s, 934 MB/s 2023-10-09T07:40:13.545 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping successful for: 2023-10-09T07:40:13.728 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:13 smithi115 ceph-mon[108015]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-09T07:40:13.904 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch daemon add osd smithi115:vg_nvme/lv_4 2023-10-09T07:40:14.121 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:40:15.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:15 smithi115 ceph-mon[108015]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-09T07:40:15.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:15 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-09T07:40:15.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:15 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-09T07:40:15.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:15 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:40:16.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:16 smithi115 ceph-mon[108015]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-09T07:40:17.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:17 smithi115 ceph-mon[108015]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-09T07:40:17.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:17 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/1074124245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b67f271-9ad5-440c-9772-cd1d602141e9"}]: dispatch 2023-10-09T07:40:17.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:17 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/1074124245' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3b67f271-9ad5-440c-9772-cd1d602141e9"}]': finished 2023-10-09T07:40:17.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:17 smithi115 ceph-mon[108015]: osdmap e5: 1 total, 0 up, 1 in 2023-10-09T07:40:17.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:17 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-09T07:40:18.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:18 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/3444369923' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-09T07:40:19.786 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:19 smithi115 ceph-mon[108015]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-09T07:40:21.847 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:21 smithi115 ceph-mon[108015]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-09T07:40:22.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:22 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-10-09T07:40:22.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:22 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:40:23.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:23 smithi115 ceph-mon[108015]: Deploying daemon osd.0 on smithi115 2023-10-09T07:40:23.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:23 smithi115 ceph-mon[108015]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-09T07:40:25.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:25 smithi115 ceph-mon[108015]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-09T07:40:25.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:25 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:25.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:25 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:25.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:25 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:40:27.455 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 0 on host 'smithi115' 2023-10-09T07:40:27.742 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:27 smithi115 ceph-mon[108015]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-09T07:40:27.742 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:27 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:27.742 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:27 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:27.742 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:27 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:40:27.743 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:27 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:40:27.743 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:27 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:27.743 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:27 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:27.743 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:27 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:27.743 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:27 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:40:27.854 DEBUG:teuthology.orchestra.run.smithi115:osd.0> sudo journalctl -f -n 0 -u ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@osd.0.service 2023-10-09T07:40:27.856 INFO:tasks.cephadm:Deploying osd.1 on smithi115 with /dev/vg_nvme/lv_3... 2023-10-09T07:40:27.856 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-10-09T07:40:27.886 INFO:journalctl@ceph.osd.0.smithi115.stdout:-- Logs begin at Mon 2023-10-09 07:30:34 UTC. -- 2023-10-09T07:40:28.405 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:40:28.852 INFO:journalctl@ceph.osd.0.smithi115.stdout:Oct 09 07:40:28 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-osd-0[118766]: 2023-10-09T07:40:28.593+0000 7f4b9278a640 -1 osd.0 0 log_to_monitors true 2023-10-09T07:40:29.687 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-10-09T07:40:29.687 INFO:teuthology.orchestra.run.smithi115.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-10-09T07:40:29.687 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10+0 records in 2023-10-09T07:40:29.688 INFO:teuthology.orchestra.run.smithi115.stderr:10+0 records out 2023-10-09T07:40:29.688 INFO:teuthology.orchestra.run.smithi115.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124512 s, 842 MB/s 2023-10-09T07:40:29.688 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping successful for: 2023-10-09T07:40:29.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:29 smithi115 ceph-mon[108015]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-09T07:40:29.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:29 smithi115 ceph-mon[108015]: from='osd.0 [v2:172.21.15.115:6802/2116278596,v1:172.21.15.115:6803/2116278596]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-10-09T07:40:30.155 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch daemon add osd smithi115:vg_nvme/lv_3 2023-10-09T07:40:30.374 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:40:30.845 INFO:journalctl@ceph.osd.0.smithi115.stdout:Oct 09 07:40:30 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-osd-0[118766]: 2023-10-09T07:40:30.479+0000 7f4b87e02700 -1 osd.0 0 waiting for initial osdmap 2023-10-09T07:40:30.845 INFO:journalctl@ceph.osd.0.smithi115.stdout:Oct 09 07:40:30 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-osd-0[118766]: 2023-10-09T07:40:30.482+0000 7f4b823f4700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-09T07:40:30.845 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:30 smithi115 ceph-mon[108015]: from='osd.0 [v2:172.21.15.115:6802/2116278596,v1:172.21.15.115:6803/2116278596]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-10-09T07:40:30.845 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:30 smithi115 ceph-mon[108015]: osdmap e6: 1 total, 0 up, 1 in 2023-10-09T07:40:30.845 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:30 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-09T07:40:30.845 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:30 smithi115 ceph-mon[108015]: from='osd.0 [v2:172.21.15.115:6802/2116278596,v1:172.21.15.115:6803/2116278596]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2023-10-09T07:40:30.846 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:30 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:30.846 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:30 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:30.846 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:30 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-09T07:40:30.846 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:30 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:30.846 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:30 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:40:30.846 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:30 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:40:30.846 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:30 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:30.846 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:30 smithi115 ceph-mon[108015]: from='osd.0 [v2:172.21.15.115:6802/2116278596,v1:172.21.15.115:6803/2116278596]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2023-10-09T07:40:30.846 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:30 smithi115 ceph-mon[108015]: osdmap e7: 1 total, 0 up, 1 in 2023-10-09T07:40:30.846 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:30 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-09T07:40:30.846 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:30 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-09T07:40:31.787 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:31 smithi115 ceph-mon[108015]: Detected new or changed devices on smithi115 2023-10-09T07:40:31.787 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:31 smithi115 ceph-mon[108015]: Adjusting osd_memory_target on smithi115 to 17113M 2023-10-09T07:40:31.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:31 smithi115 ceph-mon[108015]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-09T07:40:31.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:31 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-09T07:40:31.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:31 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-09T07:40:31.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:31 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:40:31.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:31 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-09T07:40:32.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:32 smithi115 ceph-mon[108015]: purged_snaps scrub starts 2023-10-09T07:40:32.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:32 smithi115 ceph-mon[108015]: purged_snaps scrub ok 2023-10-09T07:40:32.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:32 smithi115 ceph-mon[108015]: from='client.14197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-09T07:40:32.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:32 smithi115 ceph-mon[108015]: osd.0 [v2:172.21.15.115:6802/2116278596,v1:172.21.15.115:6803/2116278596] boot 2023-10-09T07:40:32.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:32 smithi115 ceph-mon[108015]: osdmap e8: 1 total, 1 up, 1 in 2023-10-09T07:40:32.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:32 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-09T07:40:33.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:33 smithi115 ceph-mon[108015]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-09T07:40:33.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:33 smithi115 ceph-mon[108015]: osdmap e9: 1 total, 1 up, 1 in 2023-10-09T07:40:33.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:33 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/1530671626' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "45ac536c-65da-4984-b7b4-6b1b4a16406b"}]: dispatch 2023-10-09T07:40:33.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:33 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/1530671626' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "45ac536c-65da-4984-b7b4-6b1b4a16406b"}]': finished 2023-10-09T07:40:33.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:33 smithi115 ceph-mon[108015]: osdmap e10: 2 total, 1 up, 2 in 2023-10-09T07:40:33.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:33 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-09T07:40:34.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:34 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/1755710424' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-09T07:40:35.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:35 smithi115 ceph-mon[108015]: pgmap v22: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-09T07:40:37.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:37 smithi115 ceph-mon[108015]: pgmap v23: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-09T07:40:38.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:38 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-10-09T07:40:38.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:38 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:40:39.768 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:39 smithi115 ceph-mon[108015]: pgmap v24: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-09T07:40:39.768 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:39 smithi115 ceph-mon[108015]: Deploying daemon osd.1 on smithi115 2023-10-09T07:40:41.572 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:41 smithi115 ceph-mon[108015]: pgmap v25: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-09T07:40:43.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:42 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:43.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:42 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:43.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:42 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:40:43.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:42 smithi115 ceph-mon[108015]: pgmap v26: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-09T07:40:43.534 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 1 on host 'smithi115' 2023-10-09T07:40:43.917 DEBUG:teuthology.orchestra.run.smithi115:osd.1> sudo journalctl -f -n 0 -u ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@osd.1.service 2023-10-09T07:40:43.919 INFO:tasks.cephadm:Waiting for 2 OSDs to come up... 2023-10-09T07:40:43.919 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph osd stat -f json 2023-10-09T07:40:43.952 INFO:journalctl@ceph.osd.1.smithi115.stdout:-- Logs begin at Mon 2023-10-09 07:30:34 UTC. -- 2023-10-09T07:40:43.977 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:43 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:43.977 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:43 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:43.977 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:43 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:40:43.978 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:43 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:40:43.978 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:43 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:43.978 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:43 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:43.978 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:43 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:43.978 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:43 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:40:44.425 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:40:44.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:44 smithi115 ceph-mon[108015]: pgmap v27: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-09T07:40:45.251 INFO:journalctl@ceph.osd.1.smithi115.stdout:Oct 09 07:40:44 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-osd-1[122834]: 2023-10-09T07:40:44.873+0000 7fb0e3eb1640 -1 osd.1 0 log_to_monitors true 2023-10-09T07:40:46.022 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:40:46.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:45 smithi115 ceph-mon[108015]: from='osd.1 [v2:172.21.15.115:6810/2852629843,v1:172.21.15.115:6811/2852629843]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-10-09T07:40:46.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:45 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:46.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:45 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:46.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:45 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-09T07:40:46.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:45 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-09T07:40:46.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:45 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:46.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:45 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:40:46.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:45 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:40:46.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:45 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:40:46.400 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":11,"num_osds":2,"num_up_osds":1,"osd_up_since":1696837231,"num_in_osds":2,"osd_in_since":1696837233,"num_remapped_pgs":0} 2023-10-09T07:40:47.102 INFO:journalctl@ceph.osd.1.smithi115.stdout:Oct 09 07:40:46 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-osd-1[122834]: 2023-10-09T07:40:46.851+0000 7fb0d9529700 -1 osd.1 0 waiting for initial osdmap 2023-10-09T07:40:47.102 INFO:journalctl@ceph.osd.1.smithi115.stdout:Oct 09 07:40:46 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-osd-1[122834]: 2023-10-09T07:40:46.854+0000 7fb0d531e700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-09T07:40:47.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:46 smithi115 ceph-mon[108015]: Detected new or changed devices on smithi115 2023-10-09T07:40:47.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:46 smithi115 ceph-mon[108015]: Adjusting osd_memory_target on smithi115 to 8556M 2023-10-09T07:40:47.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:46 smithi115 ceph-mon[108015]: from='osd.1 [v2:172.21.15.115:6810/2852629843,v1:172.21.15.115:6811/2852629843]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-10-09T07:40:47.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:46 smithi115 ceph-mon[108015]: osdmap e11: 2 total, 1 up, 2 in 2023-10-09T07:40:47.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:46 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-09T07:40:47.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:46 smithi115 ceph-mon[108015]: from='osd.1 [v2:172.21.15.115:6810/2852629843,v1:172.21.15.115:6811/2852629843]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2023-10-09T07:40:47.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:46 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/2416435217' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-10-09T07:40:47.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:46 smithi115 ceph-mon[108015]: pgmap v29: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-09T07:40:47.401 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph osd stat -f json 2023-10-09T07:40:47.634 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:40:48.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:47 smithi115 ceph-mon[108015]: from='osd.1 [v2:172.21.15.115:6810/2852629843,v1:172.21.15.115:6811/2852629843]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2023-10-09T07:40:48.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:47 smithi115 ceph-mon[108015]: osdmap e12: 2 total, 1 up, 2 in 2023-10-09T07:40:48.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:47 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-09T07:40:48.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:47 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-09T07:40:48.726 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:40:48.990 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:48 smithi115 ceph-mon[108015]: purged_snaps scrub starts 2023-10-09T07:40:48.990 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:48 smithi115 ceph-mon[108015]: purged_snaps scrub ok 2023-10-09T07:40:48.990 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:48 smithi115 ceph-mon[108015]: osd.1 [v2:172.21.15.115:6810/2852629843,v1:172.21.15.115:6811/2852629843] boot 2023-10-09T07:40:48.990 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:48 smithi115 ceph-mon[108015]: osdmap e13: 2 total, 2 up, 2 in 2023-10-09T07:40:48.990 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:48 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-09T07:40:48.991 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:48 smithi115 ceph-mon[108015]: pgmap v32: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:40:48.991 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:48 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/1276524171' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-10-09T07:40:49.117 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":13,"num_osds":2,"num_up_osds":2,"osd_up_since":1696837247,"num_in_osds":2,"osd_in_since":1696837233,"num_remapped_pgs":0} 2023-10-09T07:40:49.117 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph osd dump --format=json 2023-10-09T07:40:49.334 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:40:50.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:49 smithi115 ceph-mon[108015]: osdmap e14: 2 total, 2 up, 2 in 2023-10-09T07:40:50.414 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:40:50.414 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":14,"fsid":"d56ccb5c-6676-11ee-8db5-212e2dc638e7","created":"2023-10-09T07:39:04.103760+0000","modified":"2023-10-09T07:40:48.848740+0000","last_up_change":"2023-10-09T07:40:47.843497+0000","last_in_change":"2023-10-09T07:40:33.129611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":2,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3b67f271-9ad5-440c-9772-cd1d602141e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2116278596},{"type":"v1","addr":"172.21.15.115:6803","nonce":2116278596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2116278596},{"type":"v1","addr":"172.21.15.115:6805","nonce":2116278596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2116278596},{"type":"v1","addr":"172.21.15.115:6809","nonce":2116278596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2116278596},{"type":"v1","addr":"172.21.15.115:6807","nonce":2116278596}]},"public_addr":"172.21.15.115:6803/2116278596","cluster_addr":"172.21.15.115:6805/2116278596","heartbeat_back_addr":"172.21.15.115:6809/2116278596","heartbeat_front_addr":"172.21.15.115:6807/2116278596","state":["exists","up"]},{"osd":1,"uuid":"45ac536c-65da-4984-b7b4-6b1b4a16406b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":2852629843},{"type":"v1","addr":"172.21.15.115:6811","nonce":2852629843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":2852629843},{"type":"v1","addr":"172.21.15.115:6813","nonce":2852629843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":2852629843},{"type":"v1","addr":"172.21.15.115:6817","nonce":2852629843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":2852629843},{"type":"v1","addr":"172.21.15.115:6815","nonce":2852629843}]},"public_addr":"172.21.15.115:6811/2852629843","cluster_addr":"172.21.15.115:6813/2852629843","heartbeat_back_addr":"172.21.15.115:6817/2852629843","heartbeat_front_addr":"172.21.15.115:6815/2852629843","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-09T07:40:29.592709+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-09T07:40:45.862023+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2140324859":"2023-10-10T07:39:50.142678+0000","172.21.15.115:0/1936300148":"2023-10-10T07:39:50.142678+0000","172.21.15.115:6801/2140324859":"2023-10-10T07:39:50.142678+0000","172.21.15.115:0/1444659047":"2023-10-10T07:39:50.142678+0000","172.21.15.115:6801/1150240834":"2023-10-10T07:39:27.137689+0000","172.21.15.115:6800/1150240834":"2023-10-10T07:39:27.137689+0000","172.21.15.115:0/1898805328":"2023-10-10T07:39:27.137689+0000","172.21.15.115:0/1675528853":"2023-10-10T07:39:27.137689+0000","172.21.15.115:0/672898522":"2023-10-10T07:39:27.137689+0000","172.21.15.115:0/2712363277":"2023-10-10T07:39:50.142678+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-09T07:40:50.774 INFO:tasks.cephadm.ceph_manager.ceph:[] 2023-10-09T07:40:50.775 INFO:tasks.cephadm:Setting up client nodes... 2023-10-09T07:40:50.775 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-10-09T07:40:50.991 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:40:51.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:50 smithi115 ceph-mon[108015]: pgmap v34: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:40:51.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:50 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/2954578520' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-09T07:40:52.153 INFO:teuthology.orchestra.run.smithi115.stdout:[client.0] 2023-10-09T07:40:52.153 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQCEriNl/v8HCRAAFAr24eb5MkagEKuAXKEXBQ== 2023-10-09T07:40:52.481 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:52 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/2452136808' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-09T07:40:52.717 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:52 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/2452136808' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-10-09T07:40:52.721 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-10-09T07:40:52.721 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-10-09T07:40:52.721 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-09T07:40:52.762 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-09T07:40:52.765 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-10-09T07:40:52.765 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph mgr dump --format=json 2023-10-09T07:40:53.014 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:40:53.717 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:53 smithi115 ceph-mon[108015]: pgmap v35: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:40:54.173 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:40:54.581 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":13,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":540014744},{"type":"v1","addr":"172.21.15.115:6801","nonce":540014744}]},"active_addr":"172.21.15.115:6801/540014744","active_change":"2023-10-09T07:39:50.142787+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.115:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.115:0","nonce":3224727108}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.115:0","nonce":150980331}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.115:0","nonce":2391331798}]}]} 2023-10-09T07:40:55.171 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-10-09T07:40:55.171 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-10-09T07:40:55.172 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph osd dump --format=json 2023-10-09T07:40:55.173 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:54 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/1664634012' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-10-09T07:40:55.399 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:40:55.714 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:55 smithi115 ceph-mon[108015]: pgmap v36: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:40:56.531 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:40:56.531 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":14,"fsid":"d56ccb5c-6676-11ee-8db5-212e2dc638e7","created":"2023-10-09T07:39:04.103760+0000","modified":"2023-10-09T07:40:48.848740+0000","last_up_change":"2023-10-09T07:40:47.843497+0000","last_in_change":"2023-10-09T07:40:33.129611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":2,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3b67f271-9ad5-440c-9772-cd1d602141e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2116278596},{"type":"v1","addr":"172.21.15.115:6803","nonce":2116278596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2116278596},{"type":"v1","addr":"172.21.15.115:6805","nonce":2116278596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2116278596},{"type":"v1","addr":"172.21.15.115:6809","nonce":2116278596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2116278596},{"type":"v1","addr":"172.21.15.115:6807","nonce":2116278596}]},"public_addr":"172.21.15.115:6803/2116278596","cluster_addr":"172.21.15.115:6805/2116278596","heartbeat_back_addr":"172.21.15.115:6809/2116278596","heartbeat_front_addr":"172.21.15.115:6807/2116278596","state":["exists","up"]},{"osd":1,"uuid":"45ac536c-65da-4984-b7b4-6b1b4a16406b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":2852629843},{"type":"v1","addr":"172.21.15.115:6811","nonce":2852629843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":2852629843},{"type":"v1","addr":"172.21.15.115:6813","nonce":2852629843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":2852629843},{"type":"v1","addr":"172.21.15.115:6817","nonce":2852629843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":2852629843},{"type":"v1","addr":"172.21.15.115:6815","nonce":2852629843}]},"public_addr":"172.21.15.115:6811/2852629843","cluster_addr":"172.21.15.115:6813/2852629843","heartbeat_back_addr":"172.21.15.115:6817/2852629843","heartbeat_front_addr":"172.21.15.115:6815/2852629843","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-09T07:40:29.592709+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-09T07:40:45.862023+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2140324859":"2023-10-10T07:39:50.142678+0000","172.21.15.115:0/1936300148":"2023-10-10T07:39:50.142678+0000","172.21.15.115:6801/2140324859":"2023-10-10T07:39:50.142678+0000","172.21.15.115:0/1444659047":"2023-10-10T07:39:50.142678+0000","172.21.15.115:6801/1150240834":"2023-10-10T07:39:27.137689+0000","172.21.15.115:6800/1150240834":"2023-10-10T07:39:27.137689+0000","172.21.15.115:0/1898805328":"2023-10-10T07:39:27.137689+0000","172.21.15.115:0/1675528853":"2023-10-10T07:39:27.137689+0000","172.21.15.115:0/672898522":"2023-10-10T07:39:27.137689+0000","172.21.15.115:0/2712363277":"2023-10-10T07:39:50.142678+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-09T07:40:56.906 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-10-09T07:40:56.907 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph osd dump --format=json 2023-10-09T07:40:57.121 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:40:57.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:57 smithi115 ceph-mon[108015]: pgmap v37: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:40:57.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:57 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/1883263636' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-09T07:40:58.271 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:40:58.271 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":14,"fsid":"d56ccb5c-6676-11ee-8db5-212e2dc638e7","created":"2023-10-09T07:39:04.103760+0000","modified":"2023-10-09T07:40:48.848740+0000","last_up_change":"2023-10-09T07:40:47.843497+0000","last_in_change":"2023-10-09T07:40:33.129611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":2,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3b67f271-9ad5-440c-9772-cd1d602141e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2116278596},{"type":"v1","addr":"172.21.15.115:6803","nonce":2116278596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2116278596},{"type":"v1","addr":"172.21.15.115:6805","nonce":2116278596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2116278596},{"type":"v1","addr":"172.21.15.115:6809","nonce":2116278596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2116278596},{"type":"v1","addr":"172.21.15.115:6807","nonce":2116278596}]},"public_addr":"172.21.15.115:6803/2116278596","cluster_addr":"172.21.15.115:6805/2116278596","heartbeat_back_addr":"172.21.15.115:6809/2116278596","heartbeat_front_addr":"172.21.15.115:6807/2116278596","state":["exists","up"]},{"osd":1,"uuid":"45ac536c-65da-4984-b7b4-6b1b4a16406b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":2852629843},{"type":"v1","addr":"172.21.15.115:6811","nonce":2852629843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":2852629843},{"type":"v1","addr":"172.21.15.115:6813","nonce":2852629843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":2852629843},{"type":"v1","addr":"172.21.15.115:6817","nonce":2852629843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":2852629843},{"type":"v1","addr":"172.21.15.115:6815","nonce":2852629843}]},"public_addr":"172.21.15.115:6811/2852629843","cluster_addr":"172.21.15.115:6813/2852629843","heartbeat_back_addr":"172.21.15.115:6817/2852629843","heartbeat_front_addr":"172.21.15.115:6815/2852629843","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-09T07:40:29.592709+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-09T07:40:45.862023+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2140324859":"2023-10-10T07:39:50.142678+0000","172.21.15.115:0/1936300148":"2023-10-10T07:39:50.142678+0000","172.21.15.115:6801/2140324859":"2023-10-10T07:39:50.142678+0000","172.21.15.115:0/1444659047":"2023-10-10T07:39:50.142678+0000","172.21.15.115:6801/1150240834":"2023-10-10T07:39:27.137689+0000","172.21.15.115:6800/1150240834":"2023-10-10T07:39:27.137689+0000","172.21.15.115:0/1898805328":"2023-10-10T07:39:27.137689+0000","172.21.15.115:0/1675528853":"2023-10-10T07:39:27.137689+0000","172.21.15.115:0/672898522":"2023-10-10T07:39:27.137689+0000","172.21.15.115:0/2712363277":"2023-10-10T07:39:50.142678+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-09T07:40:58.571 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:58 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/914645651' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-09T07:40:58.640 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph tell osd.0 flush_pg_stats 2023-10-09T07:40:58.640 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph tell osd.1 flush_pg_stats 2023-10-09T07:40:58.865 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:40:58.915 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:40:59.715 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:40:59 smithi115 ceph-mon[108015]: pgmap v38: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:00.615 INFO:teuthology.orchestra.run.smithi115.stdout:55834574853 2023-10-09T07:41:00.615 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph osd last-stat-seq osd.1 2023-10-09T07:41:00.936 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:41:00.950 INFO:teuthology.orchestra.run.smithi115.stdout:34359738377 2023-10-09T07:41:00.950 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph osd last-stat-seq osd.0 2023-10-09T07:41:01.278 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:41:01.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:01 smithi115 ceph-mon[108015]: pgmap v39: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:02.652 INFO:teuthology.orchestra.run.smithi115.stdout:55834574853 2023-10-09T07:41:02.814 INFO:teuthology.orchestra.run.smithi115.stdout:34359738377 2023-10-09T07:41:03.038 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.1 2023-10-09T07:41:03.038 DEBUG:teuthology.parallel:result is None 2023-10-09T07:41:03.279 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738377 got 34359738377 for osd.0 2023-10-09T07:41:03.279 DEBUG:teuthology.parallel:result is None 2023-10-09T07:41:03.279 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-10-09T07:41:03.279 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph pg dump --format=json 2023-10-09T07:41:03.494 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:41:03.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:03 smithi115 ceph-mon[108015]: pgmap v40: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:03.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:03 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/272362879' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-10-09T07:41:03.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:03 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/1502322440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-10-09T07:41:04.602 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:41:04.605 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2023-10-09T07:41:04.970 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2023-10-09T07:41:04.158683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":0,"kb":187490304,"kb_used":54284,"kb_used_data":240,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":187436020,"statfs":{"total":191990071296,"available":191934484480,"internally_reserved":0,"allocated":245760,"data_stored":59464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3562,"internal_metadata":55308822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93718012,"statfs":{"total":95995035648,"available":95967244288,"internally_reserved":0,"allocated":122880,"data_stored":29732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]}]},{"osd":0,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27144,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93718008,"statfs":{"total":95995035648,"available":95967240192,"internally_reserved":0,"allocated":122880,"data_stored":29732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26100000000000001}]}]}],"pool_statfs":[]}} 2023-10-09T07:41:04.971 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph pg dump --format=json 2023-10-09T07:41:05.186 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:41:05.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:05 smithi115 ceph-mon[108015]: pgmap v41: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:06.330 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:41:06.332 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2023-10-09T07:41:06.645 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:06 smithi115 ceph-mon[108015]: from='client.14228 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:41:06.695 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2023-10-09T07:41:06.158956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":0,"kb":187490304,"kb_used":54284,"kb_used_data":240,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":187436020,"statfs":{"total":191990071296,"available":191934484480,"internally_reserved":0,"allocated":245760,"data_stored":59464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3562,"internal_metadata":55308822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93718012,"statfs":{"total":95995035648,"available":95967244288,"internally_reserved":0,"allocated":122880,"data_stored":29732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]}]},{"osd":0,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27144,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93718008,"statfs":{"total":95995035648,"available":95967240192,"internally_reserved":0,"allocated":122880,"data_stored":29732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26100000000000001}]}]}],"pool_statfs":[]}} 2023-10-09T07:41:06.695 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-10-09T07:41:06.696 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-09T07:41:06.696 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-10-09T07:41:06.696 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph health --format=json 2023-10-09T07:41:06.913 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:41:07.615 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:07 smithi115 ceph-mon[108015]: pgmap v42: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:07.616 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:07 smithi115 ceph-mon[108015]: from='client.14230 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:41:08.029 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:41:08.029 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-09T07:41:08.452 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-10-09T07:41:08.453 INFO:tasks.cephadm:Setup complete, yielding 2023-10-09T07:41:08.453 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-10-09T07:41:08.464 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi115.front.sepia.ceph.com 2023-10-09T07:41:08.465 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- bash -c 'ceph orch status' 2023-10-09T07:41:08.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:08 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/1599855205' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-10-09T07:41:08.681 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:41:09.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:09 smithi115 ceph-mon[108015]: pgmap v43: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:09.820 INFO:teuthology.orchestra.run.smithi115.stdout:Backend: cephadm 2023-10-09T07:41:09.820 INFO:teuthology.orchestra.run.smithi115.stdout:Available: Yes 2023-10-09T07:41:09.820 INFO:teuthology.orchestra.run.smithi115.stdout:Paused: No 2023-10-09T07:41:10.186 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- bash -c 'ceph orch ps' 2023-10-09T07:41:10.399 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:41:11.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:11 smithi115 ceph-mon[108015]: from='client.14234 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-10-09T07:41:11.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:11 smithi115 ceph-mon[108015]: pgmap v44: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:11.673 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-10-09T07:41:11.674 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi115 *:9283,8765 running (2m) - 2m 478M - 18.0.0-5817-gbcea8c48 fa922c444906 68ca4335fe8d 2023-10-09T07:41:11.674 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2m) - 2m 45.7M 2048M 18.0.0-5817-gbcea8c48 fa922c444906 020fdc55563b 2023-10-09T07:41:11.674 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (46s) - 46s 28.5M 8556M 18.0.0-5817-gbcea8c48 fa922c444906 53785cf795b6 2023-10-09T07:41:11.674 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (30s) - 30s 11.4M 8556M 18.0.0-5817-gbcea8c48 fa922c444906 3604be9a45b9 2023-10-09T07:41:12.061 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- bash -c 'ceph orch ls' 2023-10-09T07:41:12.277 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:41:13.436 INFO:teuthology.orchestra.run.smithi115.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-10-09T07:41:13.436 INFO:teuthology.orchestra.run.smithi115.stdout:mgr 1/1 - 62s smithi115=x;count:1 2023-10-09T07:41:13.436 INFO:teuthology.orchestra.run.smithi115.stdout:mon 1/1 - 68s smithi115:172.21.15.115=a;count:1 2023-10-09T07:41:13.436 INFO:teuthology.orchestra.run.smithi115.stdout:osd 2 - - 2023-10-09T07:41:13.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:13 smithi115 ceph-mon[108015]: from='client.14236 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-10-09T07:41:13.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:13 smithi115 ceph-mon[108015]: pgmap v45: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:13.836 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- bash -c 'ceph orch host ls' 2023-10-09T07:41:14.050 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:41:14.787 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:14 smithi115 ceph-mon[108015]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-09T07:41:15.180 INFO:teuthology.orchestra.run.smithi115.stdout:HOST ADDR LABELS STATUS 2023-10-09T07:41:15.180 INFO:teuthology.orchestra.run.smithi115.stdout:smithi115 172.21.15.115 2023-10-09T07:41:15.180 INFO:teuthology.orchestra.run.smithi115.stdout:1 hosts in cluster 2023-10-09T07:41:15.544 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- bash -c 'ceph orch device ls' 2023-10-09T07:41:15.758 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:41:15.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:15 smithi115 ceph-mon[108015]: pgmap v46: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:16.523 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:16 smithi115 ceph-mon[108015]: from='client.14240 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-09T07:41:16.867 INFO:teuthology.orchestra.run.smithi115.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-10-09T07:41:16.867 INFO:teuthology.orchestra.run.smithi115.stdout:smithi115 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400UU400BGN 372G No 31s ago LVM detected, locked 2023-10-09T07:41:17.261 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- bash -c 'ceph osd lspools' 2023-10-09T07:41:17.478 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:41:17.757 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:17 smithi115 ceph-mon[108015]: pgmap v47: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:18.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:18 smithi115 ceph-mon[108015]: from='client.14242 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-09T07:41:19.008 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- bash -c 'ceph osd pool create mypool' 2023-10-09T07:41:19.224 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:41:19.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:19 smithi115 ceph-mon[108015]: pgmap v48: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:19.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:19 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/1812182004' entity='client.admin' cmd=[{"prefix": "osd lspools"}]: dispatch 2023-10-09T07:41:20.670 INFO:teuthology.orchestra.run.smithi115.stderr:pool 'mypool' created 2023-10-09T07:41:20.991 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:20 smithi115 ceph-mon[108015]: pgmap v49: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:20.991 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:20 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/3680477992' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "mypool"}]: dispatch 2023-10-09T07:41:21.059 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- bash -c 'ceph osd pool application enable mypool rbd' 2023-10-09T07:41:21.272 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:41:22.021 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:21 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/3680477992' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "mypool"}]': finished 2023-10-09T07:41:22.022 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:21 smithi115 ceph-mon[108015]: osdmap e15: 2 total, 2 up, 2 in 2023-10-09T07:41:22.675 INFO:teuthology.orchestra.run.smithi115.stderr:enabled application 'rbd' on pool 'mypool' 2023-10-09T07:41:22.999 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:22 smithi115 ceph-mon[108015]: osdmap e16: 2 total, 2 up, 2 in 2023-10-09T07:41:22.999 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:22 smithi115 ceph-mon[108015]: pgmap v52: 32 pgs: 32 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:23.000 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:22 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/1002286483' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "mypool", "app": "rbd"}]: dispatch 2023-10-09T07:41:23.080 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- bash -c 'ceph osd lspools' 2023-10-09T07:41:23.296 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:41:23.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:23 smithi115 ceph-mon[108015]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-09T07:41:23.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:23 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/1002286483' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "mypool", "app": "rbd"}]': finished 2023-10-09T07:41:23.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:23 smithi115 ceph-mon[108015]: osdmap e17: 2 total, 2 up, 2 in 2023-10-09T07:41:24.477 INFO:teuthology.orchestra.run.smithi115.stdout:1 mypool 2023-10-09T07:41:24.782 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:24 smithi115 ceph-mon[108015]: pgmap v54: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:24.782 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:24 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/1536677825' entity='client.admin' cmd=[{"prefix": "osd lspools"}]: dispatch 2023-10-09T07:41:24.866 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- bash -c 'ceph osd pool ls detail' 2023-10-09T07:41:25.084 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:41:26.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:25 smithi115 ceph-mon[108015]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2023-10-09T07:41:26.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:25 smithi115 ceph-mon[108015]: Cluster is now healthy 2023-10-09T07:41:26.234 INFO:teuthology.orchestra.run.smithi115.stdout:pool 1 'mypool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 17 flags hashpspool stripe_width 0 application rbd read_balance_score 1.44 2023-10-09T07:41:26.234 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:41:26.608 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- bash -c 'sleep 60' 2023-10-09T07:41:26.825 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:41:26.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:26 smithi115 ceph-mon[108015]: pgmap v55: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:26.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:26 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/440531989' entity='client.admin' cmd=[{"prefix": "osd pool ls", "detail": "detail"}]: dispatch 2023-10-09T07:41:29.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:29 smithi115 ceph-mon[108015]: pgmap v56: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:31.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:31 smithi115 ceph-mon[108015]: pgmap v57: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:33.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:33 smithi115 ceph-mon[108015]: pgmap v58: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:35.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:35 smithi115 ceph-mon[108015]: pgmap v59: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:37.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:37 smithi115 ceph-mon[108015]: pgmap v60: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:39.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:39 smithi115 ceph-mon[108015]: pgmap v61: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:41.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:41 smithi115 ceph-mon[108015]: pgmap v62: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:43.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:43 smithi115 ceph-mon[108015]: pgmap v63: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:45.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:45 smithi115 ceph-mon[108015]: pgmap v64: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:46.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:46 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:41:46.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:46 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:41:46.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:46 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:41:46.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:46 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:41:47.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:47 smithi115 ceph-mon[108015]: pgmap v65: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:49.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:49 smithi115 ceph-mon[108015]: pgmap v66: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:50.602 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:41:50 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:41:50.168+0000 7fadb40d3700 -1 calc_pg_upmaps abort due to max <= 0 2023-10-09T07:41:51.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:51 smithi115 ceph-mon[108015]: pgmap v67: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:53.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:53 smithi115 ceph-mon[108015]: pgmap v68: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:55.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:55 smithi115 ceph-mon[108015]: pgmap v69: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:57.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:57 smithi115 ceph-mon[108015]: pgmap v70: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:41:59.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:41:59 smithi115 ceph-mon[108015]: pgmap v71: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:01.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:01 smithi115 ceph-mon[108015]: pgmap v72: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:03.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:03 smithi115 ceph-mon[108015]: pgmap v73: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:05.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:05 smithi115 ceph-mon[108015]: pgmap v74: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:07.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:07 smithi115 ceph-mon[108015]: pgmap v75: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:09.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:09 smithi115 ceph-mon[108015]: pgmap v76: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:11.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:11 smithi115 ceph-mon[108015]: pgmap v77: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:13.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:13 smithi115 ceph-mon[108015]: pgmap v78: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:15.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:15 smithi115 ceph-mon[108015]: pgmap v79: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:17.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:17 smithi115 ceph-mon[108015]: pgmap v80: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:19.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:19 smithi115 ceph-mon[108015]: pgmap v81: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:21.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:21 smithi115 ceph-mon[108015]: pgmap v82: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:23.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:23 smithi115 ceph-mon[108015]: pgmap v83: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:23.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:23 smithi115 ceph-mon[108015]: Health check failed: Degraded data redundancy: 32 pgs undersized (PG_DEGRADED) 2023-10-09T07:42:25.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:25 smithi115 ceph-mon[108015]: pgmap v84: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:27.827 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:27 smithi115 ceph-mon[108015]: pgmap v85: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:27.947 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- bash -c 'ceph config set mgr mgr/cephadm/container_image_nvmeof quay.io/ceph/nvmeof:0.0.4' 2023-10-09T07:42:28.165 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:42:29.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:29 smithi115 ceph-mon[108015]: pgmap v86: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:29.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:29 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/3153096015' entity='client.admin' 2023-10-09T07:42:29.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:29 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:42:29.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:29 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:42:29.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:29 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:42:29.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:29 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:42:29.704 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- bash -c 'sleep 60' 2023-10-09T07:42:29.919 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:42:31.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:31 smithi115 ceph-mon[108015]: pgmap v87: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:33.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:33 smithi115 ceph-mon[108015]: pgmap v88: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:35.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:35 smithi115 ceph-mon[108015]: pgmap v89: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:37.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:37 smithi115 ceph-mon[108015]: pgmap v90: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:39.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:39 smithi115 ceph-mon[108015]: pgmap v91: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:41.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:41 smithi115 ceph-mon[108015]: pgmap v92: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:43.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:43 smithi115 ceph-mon[108015]: pgmap v93: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:45.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:45 smithi115 ceph-mon[108015]: pgmap v94: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:47.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:47 smithi115 ceph-mon[108015]: pgmap v95: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:49.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:49 smithi115 ceph-mon[108015]: pgmap v96: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:50.602 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:42:50 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:42:50.169+0000 7fadb40d3700 -1 calc_pg_upmaps abort due to max <= 0 2023-10-09T07:42:51.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:51 smithi115 ceph-mon[108015]: pgmap v97: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:53.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:53 smithi115 ceph-mon[108015]: pgmap v98: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:55.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:55 smithi115 ceph-mon[108015]: pgmap v99: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:57.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:57 smithi115 ceph-mon[108015]: pgmap v100: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:42:59.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:42:59 smithi115 ceph-mon[108015]: pgmap v101: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:01.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:01 smithi115 ceph-mon[108015]: pgmap v102: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:03.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:03 smithi115 ceph-mon[108015]: pgmap v103: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:05.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:05 smithi115 ceph-mon[108015]: pgmap v104: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:07.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:07 smithi115 ceph-mon[108015]: pgmap v105: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:09.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:09 smithi115 ceph-mon[108015]: pgmap v106: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:11.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:11 smithi115 ceph-mon[108015]: pgmap v107: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:13.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:13 smithi115 ceph-mon[108015]: pgmap v108: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:15.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:15 smithi115 ceph-mon[108015]: pgmap v109: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:17.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:17 smithi115 ceph-mon[108015]: pgmap v110: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:19.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:19 smithi115 ceph-mon[108015]: pgmap v111: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:21.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:21 smithi115 ceph-mon[108015]: pgmap v112: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:23.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:23 smithi115 ceph-mon[108015]: pgmap v113: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:25.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:25 smithi115 ceph-mon[108015]: pgmap v114: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:27.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:27 smithi115 ceph-mon[108015]: pgmap v115: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:29.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:29 smithi115 ceph-mon[108015]: pgmap v116: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:29.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:29 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:43:30.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:30 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:43:30.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:30 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:43:30.784 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:30 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:43:31.107 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- bash -c 'ceph config get mgr mgr/cephadm/container_image_nvmeof' 2023-10-09T07:43:31.326 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:43:31.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:31 smithi115 ceph-mon[108015]: pgmap v117: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:32.512 INFO:teuthology.orchestra.run.smithi115.stdout:quay.io/ceph/nvmeof:0.0.4 2023-10-09T07:43:32.921 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- bash -c 'ceph orch apply nvmeof mypool --placement="1 $(hostname)"' 2023-10-09T07:43:33.135 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:43:33.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:33 smithi115 ceph-mon[108015]: pgmap v118: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:33.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:33 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/1411079747' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/container_image_nvmeof"}]: dispatch 2023-10-09T07:43:34.244 INFO:teuthology.orchestra.run.smithi115.stdout:Scheduled nvmeof.mypool update... 2023-10-09T07:43:34.610 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- bash -c 'ceph orch ps --refresh' 2023-10-09T07:43:34.834 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:43:35.442 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:35 smithi115 ceph-mon[108015]: pgmap v119: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:35.443 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:35 smithi115 ceph-mon[108015]: from='client.14258 -' entity='client.admin' cmd=[{"prefix": "orch apply nvmeof", "pool": "mypool", "placement": "1 smithi115", "target": ["mon-mgr", ""]}]: dispatch 2023-10-09T07:43:35.443 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:35 smithi115 ceph-mon[108015]: Saving service nvmeof.mypool spec with placement smithi115;count:1 2023-10-09T07:43:35.443 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:35 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:43:35.443 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:35 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:43:35.443 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:35 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:43:35.443 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:35 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:43:35.443 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:35 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:43:35.443 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:35 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.nvmeof.mypool.smithi115.pkdauz", "caps": ["mon", "profile rbd", "osd", "allow all tag rbd *=*"]}]: dispatch 2023-10-09T07:43:35.443 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:35 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.mypool.smithi115.pkdauz", "caps": ["mon", "profile rbd", "osd", "allow all tag rbd *=*"]}]': finished 2023-10-09T07:43:35.443 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:35 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:43:35.444 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:35 smithi115 ceph-mon[108015]: Deploying daemon nvmeof.mypool.smithi115.pkdauz on smithi115 2023-10-09T07:43:36.134 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-10-09T07:43:36.135 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi115 *:9283,8765 running (4m) - 4m 478M - 18.0.0-5817-gbcea8c48 fa922c444906 68ca4335fe8d 2023-10-09T07:43:36.135 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4m) - 4m 45.7M 2048M 18.0.0-5817-gbcea8c48 fa922c444906 020fdc55563b 2023-10-09T07:43:36.135 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3m) - 3m 28.5M 8556M 18.0.0-5817-gbcea8c48 fa922c444906 53785cf795b6 2023-10-09T07:43:36.135 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2m) - 2m 11.4M 8556M 18.0.0-5817-gbcea8c48 fa922c444906 3604be9a45b9 2023-10-09T07:43:36.628 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- bash -c 'ceph versions' 2023-10-09T07:43:36.880 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:43:37.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:37 smithi115 ceph-mon[108015]: from='client.14260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-09T07:43:37.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:37 smithi115 ceph-mon[108015]: pgmap v120: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:38.239 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-10-09T07:43:38.239 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2023-10-09T07:43:38.239 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 18.0.0-5817-gbcea8c48 (bcea8c48517ff154f392584022efa0718f51c9f2) reef (dev)": 1 2023-10-09T07:43:38.239 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2023-10-09T07:43:38.239 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2023-10-09T07:43:38.239 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 18.0.0-5817-gbcea8c48 (bcea8c48517ff154f392584022efa0718f51c9f2) reef (dev)": 1 2023-10-09T07:43:38.240 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2023-10-09T07:43:38.240 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2023-10-09T07:43:38.240 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 18.0.0-5817-gbcea8c48 (bcea8c48517ff154f392584022efa0718f51c9f2) reef (dev)": 2 2023-10-09T07:43:38.240 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2023-10-09T07:43:38.240 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2023-10-09T07:43:38.240 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 18.0.0-5817-gbcea8c48 (bcea8c48517ff154f392584022efa0718f51c9f2) reef (dev)": 4 2023-10-09T07:43:38.240 INFO:teuthology.orchestra.run.smithi115.stdout: } 2023-10-09T07:43:38.240 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-10-09T07:43:38.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:38 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/142287388' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-10-09T07:43:39.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:39 smithi115 ceph-mon[108015]: pgmap v121: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:41.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:41 smithi115 ceph-mon[108015]: pgmap v122: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:42.394 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- bash -c 'ceph -s' 2023-10-09T07:43:42.749 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:43:43.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-mon[108015]: pgmap v123: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:44.416 INFO:teuthology.orchestra.run.smithi115.stdout: cluster: 2023-10-09T07:43:44.416 INFO:teuthology.orchestra.run.smithi115.stdout: id: d56ccb5c-6676-11ee-8db5-212e2dc638e7 2023-10-09T07:43:44.416 INFO:teuthology.orchestra.run.smithi115.stdout: health: HEALTH_WARN 2023-10-09T07:43:44.416 INFO:teuthology.orchestra.run.smithi115.stdout: Degraded data redundancy: 32 pgs undersized 2023-10-09T07:43:44.416 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:43:44.416 INFO:teuthology.orchestra.run.smithi115.stdout: services: 2023-10-09T07:43:44.416 INFO:teuthology.orchestra.run.smithi115.stdout: mon: 1 daemons, quorum a (age 4m) 2023-10-09T07:43:44.417 INFO:teuthology.orchestra.run.smithi115.stdout: mgr: x(active, since 3m) 2023-10-09T07:43:44.417 INFO:teuthology.orchestra.run.smithi115.stdout: osd: 2 osds: 2 up (since 2m), 2 in (since 3m) 2023-10-09T07:43:44.417 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:43:44.417 INFO:teuthology.orchestra.run.smithi115.stdout: data: 2023-10-09T07:43:44.417 INFO:teuthology.orchestra.run.smithi115.stdout: pools: 1 pools, 32 pgs 2023-10-09T07:43:44.417 INFO:teuthology.orchestra.run.smithi115.stdout: objects: 0 objects, 0 B 2023-10-09T07:43:44.417 INFO:teuthology.orchestra.run.smithi115.stdout: usage: 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:44.417 INFO:teuthology.orchestra.run.smithi115.stdout: pgs: 32 active+undersized 2023-10-09T07:43:44.417 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:43:44.417 INFO:teuthology.orchestra.run.smithi115.stdout: progress: 2023-10-09T07:43:44.417 INFO:teuthology.orchestra.run.smithi115.stdout: Global Recovery Event (0s) 2023-10-09T07:43:44.417 INFO:teuthology.orchestra.run.smithi115.stdout: [............................] 2023-10-09T07:43:44.418 INFO:teuthology.orchestra.run.smithi115.stdout: Updating nvmeof.mypool deployment (+1 -> 1) (0s) 2023-10-09T07:43:44.418 INFO:teuthology.orchestra.run.smithi115.stdout: [............................] 2023-10-09T07:43:44.418 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:43:44.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:44 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:43:44.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:44 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:43:44.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:44 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:43:44.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:44 smithi115 ceph-mon[108015]: Checking pool "mypool" exists for service nvmeof.mypool 2023-10-09T07:43:44.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:44 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:43:44.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:44 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:43:44.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:44 smithi115 ceph-mon[108015]: pgmap v124: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:44.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:44 smithi115 ceph-mon[108015]: from='client.? 172.21.15.115:0/45264416' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-10-09T07:43:44.966 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- bash -c 'ceph orch ls' 2023-10-09T07:43:45.196 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:43:46.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:46 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:43:46.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:46 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:43:46.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:46 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:43:46.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:46 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:43:46.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:46 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:43:46.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:46 smithi115 ceph-mon[108015]: pgmap v125: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:46.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:46 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:43:46.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:46 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:43:46.399 INFO:teuthology.orchestra.run.smithi115.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-10-09T07:43:46.400 INFO:teuthology.orchestra.run.smithi115.stdout:mgr 1/1 - 3m smithi115=x;count:1 2023-10-09T07:43:46.400 INFO:teuthology.orchestra.run.smithi115.stdout:mon 1/1 - 3m smithi115:172.21.15.115=a;count:1 2023-10-09T07:43:46.400 INFO:teuthology.orchestra.run.smithi115.stdout:nvmeof.mypool ?:4420,5500,8009 0/1 - 12s smithi115;count:1 2023-10-09T07:43:46.400 INFO:teuthology.orchestra.run.smithi115.stdout:osd 2 - - 2023-10-09T07:43:46.774 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- bash -c 'rbd create mypool/myimage --size 8Gi' 2023-10-09T07:43:46.996 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:43:47.484 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:47 smithi115 ceph-mon[108015]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-10-09T07:43:47.484 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:47 smithi115 ceph-mon[108015]: mgrmap e14: x(active, since 3m) 2023-10-09T07:43:47.484 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:47 smithi115 ceph-mon[108015]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-09T07:43:47.484 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:47 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:43:47.484 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:47 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:43:47.484 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:47 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:43:47.484 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:47 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:43:47.484 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:47 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:43:48.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:48 smithi115 ceph-mon[108015]: pgmap v126: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:48.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:48 smithi115 ceph-mon[108015]: pgmap v127: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:49.826 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:49 smithi115 ceph-mon[108015]: osdmap e18: 2 total, 2 up, 2 in 2023-10-09T07:43:49.883 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- bash -c 'rbd ls mypool' 2023-10-09T07:43:50.107 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:43:50.497 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:43:50 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:43:50.171+0000 7fadb40d3700 -1 calc_pg_upmaps abort due to max <= 0 2023-10-09T07:43:50.830 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:50 smithi115 ceph-mon[108015]: pgmap v129: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:50.830 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:50 smithi115 ceph-mon[108015]: osdmap e19: 2 total, 2 up, 2 in 2023-10-09T07:43:50.963 INFO:teuthology.orchestra.run.smithi115.stdout:myimage 2023-10-09T07:43:51.317 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- bash -c 'set -ex 2023-10-09T07:43:51.317 DEBUG:teuthology.orchestra.run.smithi115:> # 2023-10-09T07:43:51.317 DEBUG:teuthology.orchestra.run.smithi115:> cat /proc/cpuinfo 2023-10-09T07:43:51.317 DEBUG:teuthology.orchestra.run.smithi115:> ' 2023-10-09T07:43:51.534 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:43:52.369 INFO:teuthology.orchestra.run.smithi115.stderr:+ cat /proc/cpuinfo 2023-10-09T07:43:52.369 INFO:teuthology.orchestra.run.smithi115.stdout:processor : 0 2023-10-09T07:43:52.370 INFO:teuthology.orchestra.run.smithi115.stdout:vendor_id : GenuineIntel 2023-10-09T07:43:52.370 INFO:teuthology.orchestra.run.smithi115.stdout:cpu family : 6 2023-10-09T07:43:52.370 INFO:teuthology.orchestra.run.smithi115.stdout:model : 79 2023-10-09T07:43:52.370 INFO:teuthology.orchestra.run.smithi115.stdout:model name : Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz 2023-10-09T07:43:52.370 INFO:teuthology.orchestra.run.smithi115.stdout:stepping : 1 2023-10-09T07:43:52.370 INFO:teuthology.orchestra.run.smithi115.stdout:microcode : 0xb000010 2023-10-09T07:43:52.370 INFO:teuthology.orchestra.run.smithi115.stdout:cpu MHz : 3800.000 2023-10-09T07:43:52.370 INFO:teuthology.orchestra.run.smithi115.stdout:cache size : 10240 KB 2023-10-09T07:43:52.370 INFO:teuthology.orchestra.run.smithi115.stdout:physical id : 0 2023-10-09T07:43:52.370 INFO:teuthology.orchestra.run.smithi115.stdout:siblings : 8 2023-10-09T07:43:52.370 INFO:teuthology.orchestra.run.smithi115.stdout:core id : 0 2023-10-09T07:43:52.371 INFO:teuthology.orchestra.run.smithi115.stdout:cpu cores : 4 2023-10-09T07:43:52.371 INFO:teuthology.orchestra.run.smithi115.stdout:apicid : 0 2023-10-09T07:43:52.371 INFO:teuthology.orchestra.run.smithi115.stdout:initial apicid : 0 2023-10-09T07:43:52.371 INFO:teuthology.orchestra.run.smithi115.stdout:fpu : yes 2023-10-09T07:43:52.371 INFO:teuthology.orchestra.run.smithi115.stdout:fpu_exception : yes 2023-10-09T07:43:52.371 INFO:teuthology.orchestra.run.smithi115.stdout:cpuid level : 20 2023-10-09T07:43:52.371 INFO:teuthology.orchestra.run.smithi115.stdout:wp : yes 2023-10-09T07:43:52.371 INFO:teuthology.orchestra.run.smithi115.stdout:flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts 2023-10-09T07:43:52.371 INFO:teuthology.orchestra.run.smithi115.stdout:bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit mmio_stale_data 2023-10-09T07:43:52.371 INFO:teuthology.orchestra.run.smithi115.stdout:bogomips : 6999.74 2023-10-09T07:43:52.371 INFO:teuthology.orchestra.run.smithi115.stdout:clflush size : 64 2023-10-09T07:43:52.371 INFO:teuthology.orchestra.run.smithi115.stdout:cache_alignment : 64 2023-10-09T07:43:52.371 INFO:teuthology.orchestra.run.smithi115.stdout:address sizes : 46 bits physical, 48 bits virtual 2023-10-09T07:43:52.372 INFO:teuthology.orchestra.run.smithi115.stdout:power management: 2023-10-09T07:43:52.372 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:43:52.372 INFO:teuthology.orchestra.run.smithi115.stdout:processor : 1 2023-10-09T07:43:52.372 INFO:teuthology.orchestra.run.smithi115.stdout:vendor_id : GenuineIntel 2023-10-09T07:43:52.372 INFO:teuthology.orchestra.run.smithi115.stdout:cpu family : 6 2023-10-09T07:43:52.372 INFO:teuthology.orchestra.run.smithi115.stdout:model : 79 2023-10-09T07:43:52.372 INFO:teuthology.orchestra.run.smithi115.stdout:model name : Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz 2023-10-09T07:43:52.372 INFO:teuthology.orchestra.run.smithi115.stdout:stepping : 1 2023-10-09T07:43:52.372 INFO:teuthology.orchestra.run.smithi115.stdout:microcode : 0xb000010 2023-10-09T07:43:52.372 INFO:teuthology.orchestra.run.smithi115.stdout:cpu MHz : 3800.000 2023-10-09T07:43:52.372 INFO:teuthology.orchestra.run.smithi115.stdout:cache size : 10240 KB 2023-10-09T07:43:52.372 INFO:teuthology.orchestra.run.smithi115.stdout:physical id : 0 2023-10-09T07:43:52.373 INFO:teuthology.orchestra.run.smithi115.stdout:siblings : 8 2023-10-09T07:43:52.373 INFO:teuthology.orchestra.run.smithi115.stdout:core id : 1 2023-10-09T07:43:52.373 INFO:teuthology.orchestra.run.smithi115.stdout:cpu cores : 4 2023-10-09T07:43:52.373 INFO:teuthology.orchestra.run.smithi115.stdout:apicid : 2 2023-10-09T07:43:52.373 INFO:teuthology.orchestra.run.smithi115.stdout:initial apicid : 2 2023-10-09T07:43:52.373 INFO:teuthology.orchestra.run.smithi115.stdout:fpu : yes 2023-10-09T07:43:52.373 INFO:teuthology.orchestra.run.smithi115.stdout:fpu_exception : yes 2023-10-09T07:43:52.373 INFO:teuthology.orchestra.run.smithi115.stdout:cpuid level : 20 2023-10-09T07:43:52.373 INFO:teuthology.orchestra.run.smithi115.stdout:wp : yes 2023-10-09T07:43:52.373 INFO:teuthology.orchestra.run.smithi115.stdout:flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts 2023-10-09T07:43:52.373 INFO:teuthology.orchestra.run.smithi115.stdout:bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit mmio_stale_data 2023-10-09T07:43:52.373 INFO:teuthology.orchestra.run.smithi115.stdout:bogomips : 6999.74 2023-10-09T07:43:52.373 INFO:teuthology.orchestra.run.smithi115.stdout:clflush size : 64 2023-10-09T07:43:52.374 INFO:teuthology.orchestra.run.smithi115.stdout:cache_alignment : 64 2023-10-09T07:43:52.374 INFO:teuthology.orchestra.run.smithi115.stdout:address sizes : 46 bits physical, 48 bits virtual 2023-10-09T07:43:52.374 INFO:teuthology.orchestra.run.smithi115.stdout:power management: 2023-10-09T07:43:52.374 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:43:52.374 INFO:teuthology.orchestra.run.smithi115.stdout:processor : 2 2023-10-09T07:43:52.374 INFO:teuthology.orchestra.run.smithi115.stdout:vendor_id : GenuineIntel 2023-10-09T07:43:52.374 INFO:teuthology.orchestra.run.smithi115.stdout:cpu family : 6 2023-10-09T07:43:52.374 INFO:teuthology.orchestra.run.smithi115.stdout:model : 79 2023-10-09T07:43:52.374 INFO:teuthology.orchestra.run.smithi115.stdout:model name : Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz 2023-10-09T07:43:52.374 INFO:teuthology.orchestra.run.smithi115.stdout:stepping : 1 2023-10-09T07:43:52.374 INFO:teuthology.orchestra.run.smithi115.stdout:microcode : 0xb000010 2023-10-09T07:43:52.374 INFO:teuthology.orchestra.run.smithi115.stdout:cpu MHz : 3800.000 2023-10-09T07:43:52.375 INFO:teuthology.orchestra.run.smithi115.stdout:cache size : 10240 KB 2023-10-09T07:43:52.375 INFO:teuthology.orchestra.run.smithi115.stdout:physical id : 0 2023-10-09T07:43:52.375 INFO:teuthology.orchestra.run.smithi115.stdout:siblings : 8 2023-10-09T07:43:52.375 INFO:teuthology.orchestra.run.smithi115.stdout:core id : 2 2023-10-09T07:43:52.375 INFO:teuthology.orchestra.run.smithi115.stdout:cpu cores : 4 2023-10-09T07:43:52.375 INFO:teuthology.orchestra.run.smithi115.stdout:apicid : 4 2023-10-09T07:43:52.375 INFO:teuthology.orchestra.run.smithi115.stdout:initial apicid : 4 2023-10-09T07:43:52.375 INFO:teuthology.orchestra.run.smithi115.stdout:fpu : yes 2023-10-09T07:43:52.375 INFO:teuthology.orchestra.run.smithi115.stdout:fpu_exception : yes 2023-10-09T07:43:52.375 INFO:teuthology.orchestra.run.smithi115.stdout:cpuid level : 20 2023-10-09T07:43:52.375 INFO:teuthology.orchestra.run.smithi115.stdout:wp : yes 2023-10-09T07:43:52.375 INFO:teuthology.orchestra.run.smithi115.stdout:flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts 2023-10-09T07:43:52.375 INFO:teuthology.orchestra.run.smithi115.stdout:bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit mmio_stale_data 2023-10-09T07:43:52.376 INFO:teuthology.orchestra.run.smithi115.stdout:bogomips : 6999.74 2023-10-09T07:43:52.376 INFO:teuthology.orchestra.run.smithi115.stdout:clflush size : 64 2023-10-09T07:43:52.376 INFO:teuthology.orchestra.run.smithi115.stdout:cache_alignment : 64 2023-10-09T07:43:52.376 INFO:teuthology.orchestra.run.smithi115.stdout:address sizes : 46 bits physical, 48 bits virtual 2023-10-09T07:43:52.376 INFO:teuthology.orchestra.run.smithi115.stdout:power management: 2023-10-09T07:43:52.376 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:43:52.376 INFO:teuthology.orchestra.run.smithi115.stdout:processor : 3 2023-10-09T07:43:52.376 INFO:teuthology.orchestra.run.smithi115.stdout:vendor_id : GenuineIntel 2023-10-09T07:43:52.376 INFO:teuthology.orchestra.run.smithi115.stdout:cpu family : 6 2023-10-09T07:43:52.376 INFO:teuthology.orchestra.run.smithi115.stdout:model : 79 2023-10-09T07:43:52.376 INFO:teuthology.orchestra.run.smithi115.stdout:model name : Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz 2023-10-09T07:43:52.376 INFO:teuthology.orchestra.run.smithi115.stdout:stepping : 1 2023-10-09T07:43:52.376 INFO:teuthology.orchestra.run.smithi115.stdout:microcode : 0xb000010 2023-10-09T07:43:52.377 INFO:teuthology.orchestra.run.smithi115.stdout:cpu MHz : 3800.000 2023-10-09T07:43:52.377 INFO:teuthology.orchestra.run.smithi115.stdout:cache size : 10240 KB 2023-10-09T07:43:52.377 INFO:teuthology.orchestra.run.smithi115.stdout:physical id : 0 2023-10-09T07:43:52.377 INFO:teuthology.orchestra.run.smithi115.stdout:siblings : 8 2023-10-09T07:43:52.377 INFO:teuthology.orchestra.run.smithi115.stdout:core id : 3 2023-10-09T07:43:52.377 INFO:teuthology.orchestra.run.smithi115.stdout:cpu cores : 4 2023-10-09T07:43:52.377 INFO:teuthology.orchestra.run.smithi115.stdout:apicid : 6 2023-10-09T07:43:52.377 INFO:teuthology.orchestra.run.smithi115.stdout:initial apicid : 6 2023-10-09T07:43:52.377 INFO:teuthology.orchestra.run.smithi115.stdout:fpu : yes 2023-10-09T07:43:52.377 INFO:teuthology.orchestra.run.smithi115.stdout:fpu_exception : yes 2023-10-09T07:43:52.377 INFO:teuthology.orchestra.run.smithi115.stdout:cpuid level : 20 2023-10-09T07:43:52.377 INFO:teuthology.orchestra.run.smithi115.stdout:wp : yes 2023-10-09T07:43:52.377 INFO:teuthology.orchestra.run.smithi115.stdout:flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts 2023-10-09T07:43:52.378 INFO:teuthology.orchestra.run.smithi115.stdout:bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit mmio_stale_data 2023-10-09T07:43:52.378 INFO:teuthology.orchestra.run.smithi115.stdout:bogomips : 6999.74 2023-10-09T07:43:52.378 INFO:teuthology.orchestra.run.smithi115.stdout:clflush size : 64 2023-10-09T07:43:52.378 INFO:teuthology.orchestra.run.smithi115.stdout:cache_alignment : 64 2023-10-09T07:43:52.378 INFO:teuthology.orchestra.run.smithi115.stdout:address sizes : 46 bits physical, 48 bits virtual 2023-10-09T07:43:52.378 INFO:teuthology.orchestra.run.smithi115.stdout:power management: 2023-10-09T07:43:52.378 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:43:52.378 INFO:teuthology.orchestra.run.smithi115.stdout:processor : 4 2023-10-09T07:43:52.378 INFO:teuthology.orchestra.run.smithi115.stdout:vendor_id : GenuineIntel 2023-10-09T07:43:52.378 INFO:teuthology.orchestra.run.smithi115.stdout:cpu family : 6 2023-10-09T07:43:52.378 INFO:teuthology.orchestra.run.smithi115.stdout:model : 79 2023-10-09T07:43:52.378 INFO:teuthology.orchestra.run.smithi115.stdout:model name : Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz 2023-10-09T07:43:52.379 INFO:teuthology.orchestra.run.smithi115.stdout:stepping : 1 2023-10-09T07:43:52.379 INFO:teuthology.orchestra.run.smithi115.stdout:microcode : 0xb000010 2023-10-09T07:43:52.379 INFO:teuthology.orchestra.run.smithi115.stdout:cpu MHz : 2655.987 2023-10-09T07:43:52.379 INFO:teuthology.orchestra.run.smithi115.stdout:cache size : 10240 KB 2023-10-09T07:43:52.379 INFO:teuthology.orchestra.run.smithi115.stdout:physical id : 0 2023-10-09T07:43:52.379 INFO:teuthology.orchestra.run.smithi115.stdout:siblings : 8 2023-10-09T07:43:52.379 INFO:teuthology.orchestra.run.smithi115.stdout:core id : 0 2023-10-09T07:43:52.379 INFO:teuthology.orchestra.run.smithi115.stdout:cpu cores : 4 2023-10-09T07:43:52.379 INFO:teuthology.orchestra.run.smithi115.stdout:apicid : 1 2023-10-09T07:43:52.379 INFO:teuthology.orchestra.run.smithi115.stdout:initial apicid : 1 2023-10-09T07:43:52.379 INFO:teuthology.orchestra.run.smithi115.stdout:fpu : yes 2023-10-09T07:43:52.379 INFO:teuthology.orchestra.run.smithi115.stdout:fpu_exception : yes 2023-10-09T07:43:52.379 INFO:teuthology.orchestra.run.smithi115.stdout:cpuid level : 20 2023-10-09T07:43:52.380 INFO:teuthology.orchestra.run.smithi115.stdout:wp : yes 2023-10-09T07:43:52.380 INFO:teuthology.orchestra.run.smithi115.stdout:flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts 2023-10-09T07:43:52.380 INFO:teuthology.orchestra.run.smithi115.stdout:bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit mmio_stale_data 2023-10-09T07:43:52.380 INFO:teuthology.orchestra.run.smithi115.stdout:bogomips : 6999.74 2023-10-09T07:43:52.380 INFO:teuthology.orchestra.run.smithi115.stdout:clflush size : 64 2023-10-09T07:43:52.380 INFO:teuthology.orchestra.run.smithi115.stdout:cache_alignment : 64 2023-10-09T07:43:52.380 INFO:teuthology.orchestra.run.smithi115.stdout:address sizes : 46 bits physical, 48 bits virtual 2023-10-09T07:43:52.380 INFO:teuthology.orchestra.run.smithi115.stdout:power management: 2023-10-09T07:43:52.380 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:43:52.380 INFO:teuthology.orchestra.run.smithi115.stdout:processor : 5 2023-10-09T07:43:52.380 INFO:teuthology.orchestra.run.smithi115.stdout:vendor_id : GenuineIntel 2023-10-09T07:43:52.380 INFO:teuthology.orchestra.run.smithi115.stdout:cpu family : 6 2023-10-09T07:43:52.380 INFO:teuthology.orchestra.run.smithi115.stdout:model : 79 2023-10-09T07:43:52.381 INFO:teuthology.orchestra.run.smithi115.stdout:model name : Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz 2023-10-09T07:43:52.381 INFO:teuthology.orchestra.run.smithi115.stdout:stepping : 1 2023-10-09T07:43:52.381 INFO:teuthology.orchestra.run.smithi115.stdout:microcode : 0xb000010 2023-10-09T07:43:52.381 INFO:teuthology.orchestra.run.smithi115.stdout:cpu MHz : 3387.825 2023-10-09T07:43:52.381 INFO:teuthology.orchestra.run.smithi115.stdout:cache size : 10240 KB 2023-10-09T07:43:52.381 INFO:teuthology.orchestra.run.smithi115.stdout:physical id : 0 2023-10-09T07:43:52.381 INFO:teuthology.orchestra.run.smithi115.stdout:siblings : 8 2023-10-09T07:43:52.381 INFO:teuthology.orchestra.run.smithi115.stdout:core id : 1 2023-10-09T07:43:52.381 INFO:teuthology.orchestra.run.smithi115.stdout:cpu cores : 4 2023-10-09T07:43:52.381 INFO:teuthology.orchestra.run.smithi115.stdout:apicid : 3 2023-10-09T07:43:52.381 INFO:teuthology.orchestra.run.smithi115.stdout:initial apicid : 3 2023-10-09T07:43:52.381 INFO:teuthology.orchestra.run.smithi115.stdout:fpu : yes 2023-10-09T07:43:52.381 INFO:teuthology.orchestra.run.smithi115.stdout:fpu_exception : yes 2023-10-09T07:43:52.382 INFO:teuthology.orchestra.run.smithi115.stdout:cpuid level : 20 2023-10-09T07:43:52.382 INFO:teuthology.orchestra.run.smithi115.stdout:wp : yes 2023-10-09T07:43:52.382 INFO:teuthology.orchestra.run.smithi115.stdout:flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts 2023-10-09T07:43:52.382 INFO:teuthology.orchestra.run.smithi115.stdout:bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit mmio_stale_data 2023-10-09T07:43:52.382 INFO:teuthology.orchestra.run.smithi115.stdout:bogomips : 6999.74 2023-10-09T07:43:52.382 INFO:teuthology.orchestra.run.smithi115.stdout:clflush size : 64 2023-10-09T07:43:52.382 INFO:teuthology.orchestra.run.smithi115.stdout:cache_alignment : 64 2023-10-09T07:43:52.382 INFO:teuthology.orchestra.run.smithi115.stdout:address sizes : 46 bits physical, 48 bits virtual 2023-10-09T07:43:52.382 INFO:teuthology.orchestra.run.smithi115.stdout:power management: 2023-10-09T07:43:52.382 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:43:52.382 INFO:teuthology.orchestra.run.smithi115.stdout:processor : 6 2023-10-09T07:43:52.382 INFO:teuthology.orchestra.run.smithi115.stdout:vendor_id : GenuineIntel 2023-10-09T07:43:52.383 INFO:teuthology.orchestra.run.smithi115.stdout:cpu family : 6 2023-10-09T07:43:52.383 INFO:teuthology.orchestra.run.smithi115.stdout:model : 79 2023-10-09T07:43:52.383 INFO:teuthology.orchestra.run.smithi115.stdout:model name : Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz 2023-10-09T07:43:52.383 INFO:teuthology.orchestra.run.smithi115.stdout:stepping : 1 2023-10-09T07:43:52.383 INFO:teuthology.orchestra.run.smithi115.stdout:microcode : 0xb000010 2023-10-09T07:43:52.383 INFO:teuthology.orchestra.run.smithi115.stdout:cpu MHz : 3800.000 2023-10-09T07:43:52.383 INFO:teuthology.orchestra.run.smithi115.stdout:cache size : 10240 KB 2023-10-09T07:43:52.383 INFO:teuthology.orchestra.run.smithi115.stdout:physical id : 0 2023-10-09T07:43:52.383 INFO:teuthology.orchestra.run.smithi115.stdout:siblings : 8 2023-10-09T07:43:52.383 INFO:teuthology.orchestra.run.smithi115.stdout:core id : 2 2023-10-09T07:43:52.383 INFO:teuthology.orchestra.run.smithi115.stdout:cpu cores : 4 2023-10-09T07:43:52.383 INFO:teuthology.orchestra.run.smithi115.stdout:apicid : 5 2023-10-09T07:43:52.384 INFO:teuthology.orchestra.run.smithi115.stdout:initial apicid : 5 2023-10-09T07:43:52.384 INFO:teuthology.orchestra.run.smithi115.stdout:fpu : yes 2023-10-09T07:43:52.384 INFO:teuthology.orchestra.run.smithi115.stdout:fpu_exception : yes 2023-10-09T07:43:52.384 INFO:teuthology.orchestra.run.smithi115.stdout:cpuid level : 20 2023-10-09T07:43:52.384 INFO:teuthology.orchestra.run.smithi115.stdout:wp : yes 2023-10-09T07:43:52.384 INFO:teuthology.orchestra.run.smithi115.stdout:flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts 2023-10-09T07:43:52.384 INFO:teuthology.orchestra.run.smithi115.stdout:bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit mmio_stale_data 2023-10-09T07:43:52.384 INFO:teuthology.orchestra.run.smithi115.stdout:bogomips : 6999.74 2023-10-09T07:43:52.384 INFO:teuthology.orchestra.run.smithi115.stdout:clflush size : 64 2023-10-09T07:43:52.384 INFO:teuthology.orchestra.run.smithi115.stdout:cache_alignment : 64 2023-10-09T07:43:52.384 INFO:teuthology.orchestra.run.smithi115.stdout:address sizes : 46 bits physical, 48 bits virtual 2023-10-09T07:43:52.384 INFO:teuthology.orchestra.run.smithi115.stdout:power management: 2023-10-09T07:43:52.384 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:43:52.385 INFO:teuthology.orchestra.run.smithi115.stdout:processor : 7 2023-10-09T07:43:52.385 INFO:teuthology.orchestra.run.smithi115.stdout:vendor_id : GenuineIntel 2023-10-09T07:43:52.385 INFO:teuthology.orchestra.run.smithi115.stdout:cpu family : 6 2023-10-09T07:43:52.385 INFO:teuthology.orchestra.run.smithi115.stdout:model : 79 2023-10-09T07:43:52.385 INFO:teuthology.orchestra.run.smithi115.stdout:model name : Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz 2023-10-09T07:43:52.385 INFO:teuthology.orchestra.run.smithi115.stdout:stepping : 1 2023-10-09T07:43:52.385 INFO:teuthology.orchestra.run.smithi115.stdout:microcode : 0xb000010 2023-10-09T07:43:52.385 INFO:teuthology.orchestra.run.smithi115.stdout:cpu MHz : 3800.000 2023-10-09T07:43:52.385 INFO:teuthology.orchestra.run.smithi115.stdout:cache size : 10240 KB 2023-10-09T07:43:52.385 INFO:teuthology.orchestra.run.smithi115.stdout:physical id : 0 2023-10-09T07:43:52.385 INFO:teuthology.orchestra.run.smithi115.stdout:siblings : 8 2023-10-09T07:43:52.385 INFO:teuthology.orchestra.run.smithi115.stdout:core id : 3 2023-10-09T07:43:52.385 INFO:teuthology.orchestra.run.smithi115.stdout:cpu cores : 4 2023-10-09T07:43:52.386 INFO:teuthology.orchestra.run.smithi115.stdout:apicid : 7 2023-10-09T07:43:52.386 INFO:teuthology.orchestra.run.smithi115.stdout:initial apicid : 7 2023-10-09T07:43:52.386 INFO:teuthology.orchestra.run.smithi115.stdout:fpu : yes 2023-10-09T07:43:52.386 INFO:teuthology.orchestra.run.smithi115.stdout:fpu_exception : yes 2023-10-09T07:43:52.386 INFO:teuthology.orchestra.run.smithi115.stdout:cpuid level : 20 2023-10-09T07:43:52.386 INFO:teuthology.orchestra.run.smithi115.stdout:wp : yes 2023-10-09T07:43:52.386 INFO:teuthology.orchestra.run.smithi115.stdout:flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts 2023-10-09T07:43:52.386 INFO:teuthology.orchestra.run.smithi115.stdout:bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit mmio_stale_data 2023-10-09T07:43:52.386 INFO:teuthology.orchestra.run.smithi115.stdout:bogomips : 6999.74 2023-10-09T07:43:52.386 INFO:teuthology.orchestra.run.smithi115.stdout:clflush size : 64 2023-10-09T07:43:52.386 INFO:teuthology.orchestra.run.smithi115.stdout:cache_alignment : 64 2023-10-09T07:43:52.386 INFO:teuthology.orchestra.run.smithi115.stdout:address sizes : 46 bits physical, 48 bits virtual 2023-10-09T07:43:52.386 INFO:teuthology.orchestra.run.smithi115.stdout:power management: 2023-10-09T07:43:52.387 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:43:52.626 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:52 smithi115 ceph-mon[108015]: pgmap v131: 32 pgs: 32 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-09T07:43:52.708 INFO:teuthology.run_tasks:Running task exec... 2023-10-09T07:43:52.726 INFO:teuthology.task.exec:Executing custom commands... 2023-10-09T07:43:52.727 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi115.front.sepia.ceph.com 2023-10-09T07:43:52.727 DEBUG:teuthology.orchestra.run.smithi115:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2023-10-09T07:43:52.727 DEBUG:teuthology.orchestra.run.smithi115:> # 2023-10-09T07:43:52.727 DEBUG:teuthology.orchestra.run.smithi115:> # print nvmeof start up logs 2023-10-09T07:43:52.727 DEBUG:teuthology.orchestra.run.smithi115:> s=$(systemctl list-units | grep nvmeof.mypool | awk '"'"'{print $1}'"'"') 2023-10-09T07:43:52.727 DEBUG:teuthology.orchestra.run.smithi115:> echo "<---- s variable is [ $s ]---->" 2023-10-09T07:43:52.727 DEBUG:teuthology.orchestra.run.smithi115:> PAGER=cat journalctl -u $s 2023-10-09T07:43:52.727 DEBUG:teuthology.orchestra.run.smithi115:> ' 2023-10-09T07:43:52.756 INFO:teuthology.orchestra.run.smithi115.stderr:++ systemctl list-units 2023-10-09T07:43:52.757 INFO:teuthology.orchestra.run.smithi115.stderr:++ grep nvmeof.mypool 2023-10-09T07:43:52.757 INFO:teuthology.orchestra.run.smithi115.stderr:++ awk '{print $1}' 2023-10-09T07:43:52.763 INFO:teuthology.orchestra.run.smithi115.stderr:+ s=ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@nvmeof.mypool.smithi115.pkdauz.service 2023-10-09T07:43:52.764 INFO:teuthology.orchestra.run.smithi115.stdout:<---- s variable is [ ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@nvmeof.mypool.smithi115.pkdauz.service ]----> 2023-10-09T07:43:52.764 INFO:teuthology.orchestra.run.smithi115.stderr:+ echo '<---- s variable is [ ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@nvmeof.mypool.smithi115.pkdauz.service ]---->' 2023-10-09T07:43:52.764 INFO:teuthology.orchestra.run.smithi115.stderr:+ PAGER=cat 2023-10-09T07:43:52.764 INFO:teuthology.orchestra.run.smithi115.stderr:+ journalctl -u ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@nvmeof.mypool.smithi115.pkdauz.service 2023-10-09T07:43:52.767 INFO:teuthology.orchestra.run.smithi115.stdout:-- Logs begin at Mon 2023-10-09 07:30:34 UTC, end at Mon 2023-10-09 07:43:52 UTC. -- 2023-10-09T07:43:52.767 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:35 smithi115 systemd[1]: Starting Ceph nvmeof.mypool.smithi115.pkdauz for d56ccb5c-6676-11ee-8db5-212e2dc638e7... 2023-10-09T07:43:52.767 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:35 smithi115 bash[130202]: Trying to pull quay.io/ceph/nvmeof:0.0.4... 2023-10-09T07:43:52.768 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:35 smithi115 bash[130202]: Getting image source signatures 2023-10-09T07:43:52.768 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:35 smithi115 bash[130202]: Copying blob sha256:eebbe46e45b08b31a85507360f62347654f4aa015db6be2f5f4c2a88d5682a6e 2023-10-09T07:43:52.768 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:35 smithi115 bash[130202]: Copying blob sha256:25d747802a0706842556a58933444a4c7149b68f8fd06d668ecf23e5c1c4d910 2023-10-09T07:43:52.768 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:35 smithi115 bash[130202]: Copying blob sha256:6f91cbbefaee416d54267a2675a2db1b0d6cdd6519d88c9feb730ee03a886724 2023-10-09T07:43:52.768 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:35 smithi115 bash[130202]: Copying blob sha256:d51fcef0a171e19c2af956d192062389670c4b89241c2d07ce2ac69a8ce260b9 2023-10-09T07:43:52.768 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:35 smithi115 bash[130202]: Copying blob sha256:90a5c35bd6aa4b341855098cc815539a9221827893281ecbcd733130dfe5de68 2023-10-09T07:43:52.768 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:35 smithi115 bash[130202]: Copying blob sha256:9fbc7479b4e92064608c6a6e79141ccbc5cba472547453eec94ff6cbffe3687d 2023-10-09T07:43:52.768 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:36 smithi115 bash[130202]: Copying blob sha256:a09b706902c6fcefd5a5a19b2370b63869630b3d054247c34b7ca1d6a9bc56e5 2023-10-09T07:43:52.768 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:42 smithi115 bash[130202]: Copying config sha256:dcfe2632d5191883d34615e3e9c69f10e215b526dfad1c4ff6bba8e4de3810d1 2023-10-09T07:43:52.768 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:42 smithi115 bash[130202]: Writing manifest to image destination 2023-10-09T07:43:52.768 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 bash[130202]: d337c51d97014662ed3553821a5f66d734ae8e2b03e4d594df80d56dd2a92c98 2023-10-09T07:43:52.768 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 systemd[1]: Started Ceph nvmeof.mypool.smithi115.pkdauz for d56ccb5c-6676-11ee-8db5-212e2dc638e7. 2023-10-09T07:43:52.769 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: INFO:control.server:Starting gateway client.nvmeof.mypool.smithi115.pkdauz 2023-10-09T07:43:52.769 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: DEBUG:control.server:Starting serve 2023-10-09T07:43:52.769 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: DEBUG:control.server:Configuring server client.nvmeof.mypool.smithi115.pkdauz 2023-10-09T07:43:52.769 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: INFO:control.server:SPDK Target Path: /usr/local/bin/nvmf_tgt 2023-10-09T07:43:52.769 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: INFO:control.server:SPDK Socket: /var/tmp/spdk.sock 2023-10-09T07:43:52.769 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: INFO:control.server:Starting /usr/local/bin/nvmf_tgt -u -r /var/tmp/spdk.sock 2023-10-09T07:43:52.769 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: INFO:control.server:Attempting to initialize SPDK: rpc_socket: /var/tmp/spdk.sock, conn_retries: 300, timeout: 60.0 2023-10-09T07:43:52.769 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: INFO: Setting log level to WARN 2023-10-09T07:43:52.769 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: INFO:JSONRPCClient(/var/tmp/spdk.sock):Setting log level to WARN 2023-10-09T07:43:52.769 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: ERROR:control.server:GatewayServer: GSIGCHLD received signum=17 2023-10-09T07:43:52.769 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: ERROR:control.server:GatewayServer exception occurred: 2023-10-09T07:43:52.769 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: Traceback (most recent call last): 2023-10-09T07:43:52.769 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: File "/usr/lib/python3.9/site-packages/spdk/rpc/client.py", line 94, in _connect 2023-10-09T07:43:52.770 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: raise socket.error("Invalid or non-existing address: '%s'" % addr) 2023-10-09T07:43:52.770 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: OSError: Invalid or non-existing address: '/var/tmp/spdk.sock' 2023-10-09T07:43:52.770 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: 2023-10-09T07:43:52.770 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: During handling of the above exception, another exception occurred: 2023-10-09T07:43:52.770 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: 2023-10-09T07:43:52.770 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: Traceback (most recent call last): 2023-10-09T07:43:52.770 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: File "/usr/lib/python3.9/site-packages/spdk/rpc/client.py", line 60, in __init__ 2023-10-09T07:43:52.770 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: self._connect(addr, port) 2023-10-09T07:43:52.770 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: File "/usr/lib/python3.9/site-packages/spdk/rpc/client.py", line 96, in _connect 2023-10-09T07:43:52.770 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: raise JSONRPCException("Error while connecting to %s\n" 2023-10-09T07:43:52.770 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: spdk.rpc.client.JSONRPCException: Error while connecting to /var/tmp/spdk.sock 2023-10-09T07:43:52.770 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: Is SPDK application running? 2023-10-09T07:43:52.770 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: Error details: Invalid or non-existing address: '/var/tmp/spdk.sock' 2023-10-09T07:43:52.771 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: 2023-10-09T07:43:52.771 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: During handling of the above exception, another exception occurred: 2023-10-09T07:43:52.771 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: 2023-10-09T07:43:52.771 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: Traceback (most recent call last): 2023-10-09T07:43:52.771 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: File "/src/control/__main__.py", line 35, in 2023-10-09T07:43:52.771 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: gateway.serve() 2023-10-09T07:43:52.771 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: File "/src/control/server.py", line 96, in serve 2023-10-09T07:43:52.771 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: self._start_spdk() 2023-10-09T07:43:52.771 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: File "/src/control/server.py", line 192, in _start_spdk 2023-10-09T07:43:52.771 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: self.spdk_rpc_client = rpc_client.JSONRPCClient( 2023-10-09T07:43:52.771 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: File "/usr/lib/python3.9/site-packages/spdk/rpc/client.py", line 64, in __init__ 2023-10-09T07:43:52.771 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: time.sleep(0.2) 2023-10-09T07:43:52.772 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: File "/src/control/server.py", line 46, in sigchld_handler 2023-10-09T07:43:52.772 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: raise SystemExit(f"spdk subprocess terminated {pid=} {exit_code=}") 2023-10-09T07:43:52.772 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: SystemExit: spdk subprocess terminated pid=3 exit_code=-4 2023-10-09T07:43:52.772 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: INFO:control.server:Aborting SPDK(client.nvmeof.mypool.smithi115.pkdauz) pid 3... 2023-10-09T07:43:52.772 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: INFO:control.server:Exiting the gateway process. 2023-10-09T07:43:52.772 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:43 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-nvmeof-mypool-smithi115-pkdauz[130626]: spdk subprocess terminated pid=3 exit_code=-4 2023-10-09T07:43:52.772 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:44 smithi115 systemd[1]: ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@nvmeof.mypool.smithi115.pkdauz.service: Main process exited, code=exited, status=1/FAILURE 2023-10-09T07:43:52.772 INFO:teuthology.orchestra.run.smithi115.stdout:Oct 09 07:43:44 smithi115 systemd[1]: ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@nvmeof.mypool.smithi115.pkdauz.service: Failed with result 'exit-code'. 2023-10-09T07:43:52.773 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2023-10-09T07:43:52.820 INFO:tasks.cephadm:Waiting for ceph service nvmeof.mypool to start (timeout 300)... 2023-10-09T07:43:52.820 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:43:53.033 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:43:53.778 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:53 smithi115 ceph-mon[108015]: pgmap v132: 32 pgs: 1 active+undersized+degraded, 31 active+undersized; 546 B data, 53 MiB used, 179 GiB / 179 GiB avail; 3/9 objects degraded (33.333%) 2023-10-09T07:43:53.779 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:53 smithi115 ceph-mon[108015]: Health check update: Degraded data redundancy: 3/9 objects degraded (33.333%), 1 pg degraded, 32 pgs undersized (PG_DEGRADED) 2023-10-09T07:43:54.156 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:43:54.156 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:43:54.550 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:43:54.815 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:54 smithi115 ceph-mon[108015]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:43:55.551 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:43:55.774 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:43:55.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:55 smithi115 ceph-mon[108015]: pgmap v133: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 0 B/s rd, 1.5 KiB/s wr, 2 op/s; 5/15 objects degraded (33.333%) 2023-10-09T07:43:56.965 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:43:56.965 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:43:57.391 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:43:58.392 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:43:58.624 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:43:58.733 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:58 smithi115 ceph-mon[108015]: pgmap v134: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 0 B/s rd, 767 B/s wr, 1 op/s; 5/15 objects degraded (33.333%) 2023-10-09T07:43:58.734 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:43:58 smithi115 ceph-mon[108015]: from='client.14276 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:43:59.759 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:43:59.760 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:44:00.249 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:44:00.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:00 smithi115 ceph-mon[108015]: pgmap v135: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 0 B/s rd, 511 B/s wr, 0 op/s; 5/15 objects degraded (33.333%) 2023-10-09T07:44:01.250 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:44:01.482 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:44:01.591 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:01 smithi115 ceph-mon[108015]: from='client.14278 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:44:02.587 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:44:02.587 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:44:02.763 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:02 smithi115 ceph-mon[108015]: pgmap v136: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 0 B/s rd, 383 B/s wr, 0 op/s; 5/15 objects degraded (33.333%) 2023-10-09T07:44:02.763 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:02 smithi115 ceph-mon[108015]: Health check update: Degraded data redundancy: 5/15 objects degraded (33.333%), 2 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2023-10-09T07:44:02.990 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:44:03.991 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:44:04.219 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:44:04.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:04 smithi115 ceph-mon[108015]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:44:04.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:04 smithi115 ceph-mon[108015]: pgmap v137: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 0 B/s rd, 307 B/s wr, 0 op/s; 5/15 objects degraded (33.333%) 2023-10-09T07:44:05.353 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:44:05.353 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:44:05.777 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:44:06.703 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:06 smithi115 ceph-mon[108015]: pgmap v138: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 0 B/s rd, 255 B/s wr, 0 op/s; 5/15 objects degraded (33.333%) 2023-10-09T07:44:06.703 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:06 smithi115 ceph-mon[108015]: from='client.14282 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:44:06.778 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:44:06.998 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:44:08.138 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:44:08.139 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:44:08.555 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:44:08.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:08 smithi115 ceph-mon[108015]: pgmap v139: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:09.556 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:44:09.788 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:44:09.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:09 smithi115 ceph-mon[108015]: from='client.14284 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:44:10.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:10 smithi115 ceph-mon[108015]: pgmap v140: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:12.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:12 smithi115 ceph-mon[108015]: pgmap v141: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:13.272 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:44:13.272 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:44:14.762 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:14 smithi115 ceph-mon[108015]: pgmap v142: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:14.762 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:14 smithi115 ceph-mon[108015]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:44:14.786 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:44:15.788 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:44:16.018 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:44:16.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:16 smithi115 ceph-mon[108015]: pgmap v143: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:17.157 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:44:17.157 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:44:17.661 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:44:18.662 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:44:18.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:18 smithi115 ceph-mon[108015]: pgmap v144: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:18.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:18 smithi115 ceph-mon[108015]: from='client.14288 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:44:18.896 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:44:20.356 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:44:20.356 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:44:20.693 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:20 smithi115 ceph-mon[108015]: pgmap v145: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:20.727 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:44:21.728 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:44:21.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:21 smithi115 ceph-mon[108015]: from='client.14290 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:44:21.961 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:44:22.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:22 smithi115 ceph-mon[108015]: pgmap v146: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:23.145 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:44:23.145 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:44:23.518 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:44:24.519 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:44:24.746 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:44:24.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:24 smithi115 ceph-mon[108015]: pgmap v147: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:24.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:24 smithi115 ceph-mon[108015]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:44:25.913 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:44:25.914 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:44:26.309 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:44:26.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:26 smithi115 ceph-mon[108015]: pgmap v148: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:27.311 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:44:27.540 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:44:27.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:27 smithi115 ceph-mon[108015]: from='client.14294 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:44:28.813 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:28 smithi115 ceph-mon[108015]: pgmap v149: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:28.951 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:44:28.952 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:44:29.352 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:44:30.354 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:44:30.589 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:44:30.702 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:30 smithi115 ceph-mon[108015]: pgmap v150: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:30.702 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:30 smithi115 ceph-mon[108015]: from='client.14296 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:44:31.712 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:44:31.712 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:44:32.083 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:44:32.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:32 smithi115 ceph-mon[108015]: pgmap v151: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:33.085 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:44:33.315 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:44:33.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:33 smithi115 ceph-mon[108015]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:44:34.450 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:44:34.450 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:44:34.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:34 smithi115 ceph-mon[108015]: pgmap v152: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:34.824 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:44:35.826 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:44:35.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:35 smithi115 ceph-mon[108015]: from='client.14300 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:44:36.054 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:44:36.812 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:36 smithi115 ceph-mon[108015]: pgmap v153: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:37.208 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:44:37.208 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:44:37.582 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:44:38.583 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:44:38.813 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:44:38.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:38 smithi115 ceph-mon[108015]: pgmap v154: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:38.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:38 smithi115 ceph-mon[108015]: from='client.14302 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:44:39.959 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:44:39.959 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:44:40.314 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:44:40.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:40 smithi115 ceph-mon[108015]: pgmap v155: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:41.315 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:44:41.546 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:44:41.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:41 smithi115 ceph-mon[108015]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:44:42.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:42 smithi115 ceph-mon[108015]: pgmap v156: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:42.714 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:44:42.714 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:44:43.091 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:44:44.091 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:44:44.322 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:44:44.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:44 smithi115 ceph-mon[108015]: from='client.14306 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:44:44.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:44 smithi115 ceph-mon[108015]: pgmap v157: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:45.480 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:44:45.480 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:44:46.328 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:44:46.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:46 smithi115 ceph-mon[108015]: pgmap v158: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:46.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:46 smithi115 ceph-mon[108015]: from='client.14308 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:44:47.329 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:44:47.543 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:44:47.658 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:47 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:44:47.658 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:47 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:44:47.658 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:47 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:44:47.658 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:47 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:44:48.626 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:44:48.626 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:44:48.808 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:48 smithi115 ceph-mon[108015]: pgmap v159: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:48.808 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:48 smithi115 ceph-mon[108015]: pgmap v160: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:48.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:48 smithi115 ceph-mon[108015]: pgmap v161: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:49.021 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:44:50.022 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:44:50.255 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:44:50.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:50 smithi115 ceph-mon[108015]: from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:44:50.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:50 smithi115 ceph-mon[108015]: pgmap v162: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:51.396 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:44:51.396 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:44:51.778 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:44:52.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:51 smithi115 ceph-mon[108015]: pgmap v163: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:52.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:51 smithi115 ceph-mon[108015]: from='client.14312 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:44:52.779 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:44:53.011 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:44:54.138 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:44:54.138 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:44:54.519 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:44:54.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:54 smithi115 ceph-mon[108015]: pgmap v164: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:55.520 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:44:55.749 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:44:55.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:55 smithi115 ceph-mon[108015]: from='client.14314 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:44:56.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:56 smithi115 ceph-mon[108015]: pgmap v165: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:56.833 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:44:56.833 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:44:57.211 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:44:58.212 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:44:58.442 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:44:58.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:58 smithi115 ceph-mon[108015]: from='client.14316 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:44:58.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:44:58 smithi115 ceph-mon[108015]: pgmap v166: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:44:59.557 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:44:59.557 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:44:59.918 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:45:00.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:00 smithi115 ceph-mon[108015]: pgmap v167: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:00.919 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:45:01.150 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:45:01.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:01 smithi115 ceph-mon[108015]: from='client.14318 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:45:02.379 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:45:02.379 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:45:02.751 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:45:02.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:02 smithi115 ceph-mon[108015]: pgmap v168: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:03.753 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:45:03.786 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:03 smithi115 ceph-mon[108015]: from='client.14320 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:45:03.983 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:45:04.726 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:04 smithi115 ceph-mon[108015]: pgmap v169: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:05.082 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:45:05.082 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:45:05.458 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:45:06.459 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:45:06.692 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:45:06.831 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:06 smithi115 ceph-mon[108015]: from='client.14322 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:45:06.831 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:06 smithi115 ceph-mon[108015]: pgmap v170: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:07.868 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:45:07.868 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:45:08.252 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:45:08.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:08 smithi115 ceph-mon[108015]: pgmap v171: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:09.253 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:45:09.483 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:45:09.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:09 smithi115 ceph-mon[108015]: from='client.14324 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:45:10.644 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:45:10.644 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:45:10.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:10 smithi115 ceph-mon[108015]: pgmap v172: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:11.011 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:45:12.012 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:45:12.242 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:45:12.839 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:12 smithi115 ceph-mon[108015]: from='client.14326 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:45:12.840 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:12 smithi115 ceph-mon[108015]: pgmap v173: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:13.431 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:45:13.431 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:45:13.815 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:45:14.817 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:45:14.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:14 smithi115 ceph-mon[108015]: pgmap v174: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:14.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:14 smithi115 ceph-mon[108015]: from='client.14328 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:45:15.049 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:45:16.236 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:45:16.236 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:45:16.614 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:45:16.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:16 smithi115 ceph-mon[108015]: pgmap v175: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:17.615 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:45:17.844 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:45:17.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:17 smithi115 ceph-mon[108015]: from='client.14330 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:45:18.644 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:18 smithi115 ceph-mon[108015]: pgmap v176: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:19.026 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:45:19.026 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:45:19.390 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:45:20.391 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:45:20.622 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:45:20.772 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:20 smithi115 ceph-mon[108015]: from='client.14332 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:45:20.772 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:20 smithi115 ceph-mon[108015]: pgmap v177: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:21.802 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:45:21.803 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:45:22.187 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:45:22.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:22 smithi115 ceph-mon[108015]: pgmap v178: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:23.188 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:45:23.420 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:45:23.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:23 smithi115 ceph-mon[108015]: from='client.14334 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:45:24.618 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:45:24.619 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:45:24.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:24 smithi115 ceph-mon[108015]: pgmap v179: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:24.995 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:45:25.996 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:45:26.226 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:45:26.745 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:26 smithi115 ceph-mon[108015]: from='client.14336 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:45:26.745 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:26 smithi115 ceph-mon[108015]: pgmap v180: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:27.386 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:45:27.386 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:45:27.756 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:45:28.756 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:45:28.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:28 smithi115 ceph-mon[108015]: pgmap v181: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:28.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:28 smithi115 ceph-mon[108015]: from='client.14338 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:45:28.986 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:45:30.122 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:45:30.122 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:45:30.503 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:45:30.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:30 smithi115 ceph-mon[108015]: pgmap v182: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:31.505 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:45:31.739 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:45:31.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:31 smithi115 ceph-mon[108015]: from='client.14340 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:45:32.828 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:32 smithi115 ceph-mon[108015]: pgmap v183: 32 pgs: 2 active+undersized+degraded, 30 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:32.856 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:45:32.856 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:45:33.245 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:45:33.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:33 smithi115 ceph-mon[108015]: from='client.14342 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:45:33.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:33 smithi115 ceph-mon[108015]: pgmap v184: 32 pgs: 3 active+undersized+degraded, 29 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:33.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:33 smithi115 ceph-mon[108015]: Health check update: Degraded data redundancy: 5/15 objects degraded (33.333%), 3 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2023-10-09T07:45:34.245 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:45:34.476 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:45:35.606 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:45:35.606 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:45:35.968 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:45:36.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:36 smithi115 ceph-mon[108015]: pgmap v185: 32 pgs: 3 active+undersized+degraded, 29 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:36.969 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:45:37.202 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:45:37.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:37 smithi115 ceph-mon[108015]: from='client.14344 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:45:38.368 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:45:38.368 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:45:38.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:38 smithi115 ceph-mon[108015]: pgmap v186: 32 pgs: 3 active+undersized+degraded, 29 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:38.752 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:45:39.753 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:45:39.785 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:39 smithi115 ceph-mon[108015]: from='client.14346 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:45:39.984 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:45:40.767 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:40 smithi115 ceph-mon[108015]: pgmap v187: 32 pgs: 3 active+undersized+degraded, 29 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:41.142 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:45:41.142 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:45:41.521 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:45:42.521 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:45:42.751 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:45:42.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:42 smithi115 ceph-mon[108015]: from='client.14348 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:45:42.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:42 smithi115 ceph-mon[108015]: pgmap v188: 32 pgs: 3 active+undersized+degraded, 29 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:43.944 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:45:43.944 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:45:44.317 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:45:44.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:44 smithi115 ceph-mon[108015]: pgmap v189: 32 pgs: 3 active+undersized+degraded, 29 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:45.318 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:45:45.549 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:45:45.668 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:45 smithi115 ceph-mon[108015]: from='client.14350 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:45:46.671 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:45:46.671 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:45:46.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:46 smithi115 ceph-mon[108015]: pgmap v190: 32 pgs: 3 active+undersized+degraded, 29 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:47.049 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:45:47.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:47 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:45:48.050 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:45:48.266 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:45:48.831 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:48 smithi115 ceph-mon[108015]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:45:48.831 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:48 smithi115 ceph-mon[108015]: pgmap v191: 32 pgs: 3 active+undersized+degraded, 29 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:48.831 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:48 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:45:48.832 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:48 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:45:48.832 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:48 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:45:49.372 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:45:49.372 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:45:49.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:49 smithi115 ceph-mon[108015]: pgmap v192: 32 pgs: 3 active+undersized+degraded, 29 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:49.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:49 smithi115 ceph-mon[108015]: pgmap v193: 32 pgs: 3 active+undersized+degraded, 29 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:49.757 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:45:50.758 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:45:50.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:50 smithi115 ceph-mon[108015]: from='client.14354 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:45:50.988 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:45:51.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:51 smithi115 ceph-mon[108015]: pgmap v194: 32 pgs: 3 active+undersized+degraded, 29 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:52.156 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:45:52.156 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:45:52.540 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:45:53.541 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:45:53.766 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:45:53.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:53 smithi115 ceph-mon[108015]: pgmap v195: 32 pgs: 3 active+undersized+degraded, 29 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:53.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:53 smithi115 ceph-mon[108015]: from='client.14356 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:45:54.917 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:45:54.918 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:45:55.358 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:45:55.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:55 smithi115 ceph-mon[108015]: pgmap v196: 32 pgs: 3 active+undersized+degraded, 29 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:56.359 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:45:56.590 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:45:56.692 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:56 smithi115 ceph-mon[108015]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:45:57.741 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:45:57.742 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:45:57.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:57 smithi115 ceph-mon[108015]: pgmap v197: 32 pgs: 3 active+undersized+degraded, 29 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:58.124 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:45:59.125 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:45:59.357 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:45:59.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:59 smithi115 ceph-mon[108015]: pgmap v198: 32 pgs: 3 active+undersized+degraded, 29 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:45:59.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:45:59 smithi115 ceph-mon[108015]: from='client.14360 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:46:00.499 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:46:00.499 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:46:00.821 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:46:01.822 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:46:01.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:01 smithi115 ceph-mon[108015]: pgmap v199: 32 pgs: 3 active+undersized+degraded, 29 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:01.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:01 smithi115 ceph-mon[108015]: from='client.14362 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:46:02.052 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:46:03.122 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:46:03.122 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:46:03.564 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:46:03.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:03 smithi115 ceph-mon[108015]: pgmap v200: 32 pgs: 3 active+undersized+degraded, 29 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:04.565 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:46:04.793 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:46:04.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:04 smithi115 ceph-mon[108015]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:46:05.623 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:05 smithi115 ceph-mon[108015]: pgmap v201: 32 pgs: 4 active+undersized+degraded, 28 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:05.624 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:05 smithi115 ceph-mon[108015]: Health check update: Degraded data redundancy: 5/15 objects degraded (33.333%), 4 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2023-10-09T07:46:05.972 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:46:05.972 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:46:06.387 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:46:07.388 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:46:07.619 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:46:07.730 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:07 smithi115 ceph-mon[108015]: pgmap v202: 32 pgs: 4 active+undersized+degraded, 28 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:07.730 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:07 smithi115 ceph-mon[108015]: from='client.14366 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:46:08.754 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:46:08.755 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:46:09.153 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:46:09.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:09 smithi115 ceph-mon[108015]: pgmap v203: 32 pgs: 4 active+undersized+degraded, 28 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:10.154 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:46:10.386 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:46:10.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:10 smithi115 ceph-mon[108015]: from='client.14368 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:46:11.528 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:46:11.529 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:46:11.748 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:11 smithi115 ceph-mon[108015]: pgmap v204: 32 pgs: 4 active+undersized+degraded, 28 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:11.961 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:46:12.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:12 smithi115 ceph-mon[108015]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:46:12.962 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:46:13.192 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:46:13.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:13 smithi115 ceph-mon[108015]: pgmap v205: 32 pgs: 4 active+undersized+degraded, 28 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:14.312 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:46:14.313 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:46:14.693 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:46:14.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:14 smithi115 ceph-mon[108015]: pgmap v206: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:14.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:14 smithi115 ceph-mon[108015]: from='client.14372 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:46:14.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:14 smithi115 ceph-mon[108015]: Health check update: Degraded data redundancy: 5/15 objects degraded (33.333%), 5 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2023-10-09T07:46:15.694 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:46:15.923 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:46:17.055 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:46:17.055 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:46:17.077 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:16 smithi115 ceph-mon[108015]: pgmap v207: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:17.468 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:46:18.469 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:46:18.701 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:46:18.844 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:18 smithi115 ceph-mon[108015]: from='client.14374 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:46:19.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:19 smithi115 ceph-mon[108015]: pgmap v208: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:19.865 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:46:19.866 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:46:20.277 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:46:21.279 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:46:21.509 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:46:21.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:21 smithi115 ceph-mon[108015]: pgmap v209: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:21.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:21 smithi115 ceph-mon[108015]: from='client.14376 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:46:22.616 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:46:22.616 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:46:23.059 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:46:23.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:23 smithi115 ceph-mon[108015]: pgmap v210: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:24.061 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:46:24.289 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:46:24.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:24 smithi115 ceph-mon[108015]: from='client.14378 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:46:25.416 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:46:25.416 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:46:25.731 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:25 smithi115 ceph-mon[108015]: pgmap v211: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:25.799 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:46:26.800 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:46:26.833 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:26 smithi115 ceph-mon[108015]: from='client.14380 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:46:27.030 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:46:27.786 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:27 smithi115 ceph-mon[108015]: pgmap v212: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:28.135 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:46:28.135 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:46:28.535 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:46:29.536 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:46:29.768 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:46:29.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:29 smithi115 ceph-mon[108015]: pgmap v213: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:29.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:29 smithi115 ceph-mon[108015]: from='client.14382 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:46:30.906 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:46:30.907 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:46:31.323 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:46:31.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:31 smithi115 ceph-mon[108015]: pgmap v214: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:32.325 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:46:32.555 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:46:32.666 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:32 smithi115 ceph-mon[108015]: from='client.14384 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:46:33.704 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:46:33.704 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:46:33.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:33 smithi115 ceph-mon[108015]: pgmap v215: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:34.123 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:46:35.124 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:46:35.354 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:46:35.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:35 smithi115 ceph-mon[108015]: from='client.14386 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:46:35.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:35 smithi115 ceph-mon[108015]: pgmap v216: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:36.480 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:46:36.480 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:46:36.889 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:46:37.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:37 smithi115 ceph-mon[108015]: pgmap v217: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:37.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:37 smithi115 ceph-mon[108015]: from='client.14388 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:46:37.890 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:46:38.119 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:46:39.268 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:46:39.268 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:46:39.671 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:46:39.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:39 smithi115 ceph-mon[108015]: pgmap v218: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:40.672 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:46:40.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:40 smithi115 ceph-mon[108015]: from='client.14390 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:46:40.904 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:46:41.681 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:41 smithi115 ceph-mon[108015]: pgmap v219: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:42.035 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:46:42.035 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:46:42.437 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:46:43.439 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:46:43.670 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:46:43.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:43 smithi115 ceph-mon[108015]: pgmap v220: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:43.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:43 smithi115 ceph-mon[108015]: from='client.14392 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:46:44.841 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:46:44.841 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:46:45.238 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:46:45.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:45 smithi115 ceph-mon[108015]: pgmap v221: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:46.240 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:46:46.470 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:46:46.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:46 smithi115 ceph-mon[108015]: from='client.14394 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:46:47.637 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:46:47.637 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:46:47.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:47 smithi115 ceph-mon[108015]: pgmap v222: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:48.022 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:46:48.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:48 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:46:48.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:48 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:46:48.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:48 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:46:48.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:48 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:46:49.023 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:46:49.241 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:46:49.776 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:49 smithi115 ceph-mon[108015]: from='client.14396 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:46:49.776 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:49 smithi115 ceph-mon[108015]: pgmap v223: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:49.776 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:49 smithi115 ceph-mon[108015]: pgmap v224: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:49.776 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:49 smithi115 ceph-mon[108015]: pgmap v225: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:50.441 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:46:50.441 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:46:50.827 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:46:51.829 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:46:51.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:51 smithi115 ceph-mon[108015]: pgmap v226: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:51.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:51 smithi115 ceph-mon[108015]: from='client.14398 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:46:52.061 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:46:53.194 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:46:53.195 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:46:53.577 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:46:53.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:53 smithi115 ceph-mon[108015]: pgmap v227: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:54.578 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:46:54.808 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:46:54.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:54 smithi115 ceph-mon[108015]: from='client.14400 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:46:55.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:55 smithi115 ceph-mon[108015]: pgmap v228: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:55.932 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:46:55.932 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:46:56.301 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:46:57.302 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:46:57.532 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:46:57.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:57 smithi115 ceph-mon[108015]: from='client.14402 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:46:57.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:57 smithi115 ceph-mon[108015]: pgmap v229: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:46:58.657 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:46:58.657 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:46:59.041 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:46:59.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:46:59 smithi115 ceph-mon[108015]: pgmap v230: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:00.043 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:47:00.276 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:47:00.799 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:00 smithi115 ceph-mon[108015]: from='client.14404 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:47:01.432 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:47:01.432 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:47:01.690 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:01 smithi115 ceph-mon[108015]: pgmap v231: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:01.807 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:47:02.808 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:47:02.842 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:02 smithi115 ceph-mon[108015]: from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:47:03.041 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:47:03.819 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:03 smithi115 ceph-mon[108015]: pgmap v232: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:04.216 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:47:04.216 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:47:04.607 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:47:05.608 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:47:05.839 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:47:05.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:05 smithi115 ceph-mon[108015]: pgmap v233: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:05.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:05 smithi115 ceph-mon[108015]: from='client.14408 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:47:06.943 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:47:06.943 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:47:07.324 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:47:07.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:07 smithi115 ceph-mon[108015]: pgmap v234: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:08.325 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:47:08.556 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:47:08.675 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:08 smithi115 ceph-mon[108015]: from='client.14410 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:47:09.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:09 smithi115 ceph-mon[108015]: pgmap v235: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:09.764 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:47:09.765 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:47:10.191 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:47:10.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:10 smithi115 ceph-mon[108015]: from='client.14412 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:47:10.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:10 smithi115 ceph-mon[108015]: pgmap v236: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:11.192 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:47:11.423 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:47:12.557 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:47:12.557 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:47:12.955 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:47:13.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:13 smithi115 ceph-mon[108015]: pgmap v237: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:13.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:13 smithi115 ceph-mon[108015]: from='client.14414 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:47:13.957 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:47:14.186 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:47:15.325 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:47:15.326 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:47:15.705 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:47:15.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:15 smithi115 ceph-mon[108015]: pgmap v238: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:16.706 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:47:16.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:16 smithi115 ceph-mon[108015]: from='client.14416 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:47:16.936 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:47:17.699 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:17 smithi115 ceph-mon[108015]: pgmap v239: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:18.116 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:47:18.116 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:47:18.496 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:47:19.497 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:47:19.727 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:47:19.822 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:19 smithi115 ceph-mon[108015]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:47:19.822 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:19 smithi115 ceph-mon[108015]: pgmap v240: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:20.798 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:47:20.799 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:47:21.187 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:47:21.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:21 smithi115 ceph-mon[108015]: pgmap v241: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:22.188 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:47:22.419 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:47:22.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:22 smithi115 ceph-mon[108015]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:47:23.526 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:47:23.526 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:47:23.843 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:23 smithi115 ceph-mon[108015]: pgmap v242: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:23.928 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:47:24.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:24 smithi115 ceph-mon[108015]: from='client.14422 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:47:24.929 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:47:25.159 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:47:25.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:25 smithi115 ceph-mon[108015]: pgmap v243: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:26.349 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:47:26.350 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:47:26.737 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:47:27.738 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:47:27.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:27 smithi115 ceph-mon[108015]: pgmap v244: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:27.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:27 smithi115 ceph-mon[108015]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:47:27.969 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:47:29.118 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:47:29.118 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:47:29.502 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:47:29.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:29 smithi115 ceph-mon[108015]: pgmap v245: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:30.503 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:47:30.735 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:47:30.845 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:30 smithi115 ceph-mon[108015]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:47:31.818 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:31 smithi115 ceph-mon[108015]: pgmap v246: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:31.851 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:47:31.851 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:47:32.234 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:47:33.235 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:47:33.465 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:47:33.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:33 smithi115 ceph-mon[108015]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:47:33.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:33 smithi115 ceph-mon[108015]: pgmap v247: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:34.606 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:47:34.606 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:47:34.993 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:47:35.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:35 smithi115 ceph-mon[108015]: pgmap v248: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:35.994 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:47:36.223 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:47:36.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:36 smithi115 ceph-mon[108015]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:47:37.383 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:47:37.384 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:47:37.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:37 smithi115 ceph-mon[108015]: pgmap v249: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:37.767 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:47:38.768 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:47:38.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:38 smithi115 ceph-mon[108015]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:47:39.001 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:47:39.770 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:39 smithi115 ceph-mon[108015]: pgmap v250: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:40.116 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:47:40.116 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:47:40.466 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:47:41.467 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:47:41.698 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:47:41.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:41 smithi115 ceph-mon[108015]: from='client.14434 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:47:41.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:41 smithi115 ceph-mon[108015]: pgmap v251: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:42.817 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:47:42.818 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:47:43.233 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:47:43.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:43 smithi115 ceph-mon[108015]: pgmap v252: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:44.233 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:47:44.463 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:47:44.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:44 smithi115 ceph-mon[108015]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:47:45.620 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:47:45.620 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:47:45.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:45 smithi115 ceph-mon[108015]: pgmap v253: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:46.074 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:47:47.075 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:47:47.305 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:47:47.763 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:47 smithi115 ceph-mon[108015]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:47:47.763 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:47 smithi115 ceph-mon[108015]: pgmap v254: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:48.425 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:47:48.426 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:47:48.738 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:48 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:47:48.814 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:47:49.815 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:47:49.847 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:49 smithi115 ceph-mon[108015]: pgmap v255: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:49.847 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:49 smithi115 ceph-mon[108015]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:47:49.847 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:49 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:47:49.847 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:49 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:47:49.847 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:49 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:47:50.046 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:47:50.818 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:50 smithi115 ceph-mon[108015]: pgmap v256: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:50.818 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:50 smithi115 ceph-mon[108015]: pgmap v257: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:51.240 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:47:51.240 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:47:51.629 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:47:51.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:51 smithi115 ceph-mon[108015]: pgmap v258: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:51.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:51 smithi115 ceph-mon[108015]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:47:52.630 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:47:52.861 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:47:53.957 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:47:53.957 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:47:54.338 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:47:54.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:54 smithi115 ceph-mon[108015]: pgmap v259: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:55.339 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:47:55.568 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:47:55.672 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:55 smithi115 ceph-mon[108015]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:47:56.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:56 smithi115 ceph-mon[108015]: pgmap v260: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:56.668 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:47:56.668 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:47:57.088 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:47:58.089 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:47:58.321 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:47:58.835 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:58 smithi115 ceph-mon[108015]: pgmap v261: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:47:58.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:47:58 smithi115 ceph-mon[108015]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:47:59.461 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:47:59.462 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:47:59.849 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:48:00.850 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:48:00.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:00 smithi115 ceph-mon[108015]: pgmap v262: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:00.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:00 smithi115 ceph-mon[108015]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:48:01.079 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:48:02.221 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:48:02.221 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:48:02.610 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:48:02.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:02 smithi115 ceph-mon[108015]: pgmap v263: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:03.611 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:48:03.843 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:48:03.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:03 smithi115 ceph-mon[108015]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:48:04.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:04 smithi115 ceph-mon[108015]: pgmap v264: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:04.981 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:48:04.981 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:48:05.402 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:48:06.403 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:48:06.633 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:48:06.736 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:06 smithi115 ceph-mon[108015]: pgmap v265: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:06.736 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:06 smithi115 ceph-mon[108015]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:48:07.778 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:48:07.778 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:48:08.185 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:48:08.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:08 smithi115 ceph-mon[108015]: pgmap v266: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:09.186 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:48:09.418 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:48:09.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:09 smithi115 ceph-mon[108015]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:48:10.526 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:48:10.527 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:48:10.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:10 smithi115 ceph-mon[108015]: pgmap v267: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:10.917 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:48:11.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:11 smithi115 ceph-mon[108015]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:48:11.917 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:48:12.147 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:48:12.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:12 smithi115 ceph-mon[108015]: pgmap v268: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:13.311 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:48:13.312 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:48:13.700 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:48:14.701 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:48:14.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:14 smithi115 ceph-mon[108015]: pgmap v269: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:14.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:14 smithi115 ceph-mon[108015]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:48:14.931 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:48:16.128 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:48:16.128 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:48:16.500 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:48:16.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:16 smithi115 ceph-mon[108015]: pgmap v270: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:17.501 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:48:17.734 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:48:17.842 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:17 smithi115 ceph-mon[108015]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:48:18.815 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:18 smithi115 ceph-mon[108015]: pgmap v271: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:18.894 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:48:18.894 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:48:19.283 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:48:20.284 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:48:20.514 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:48:20.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:20 smithi115 ceph-mon[108015]: pgmap v272: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:20.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:20 smithi115 ceph-mon[108015]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:48:21.632 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:48:21.772 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:48:22.040 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:48:22.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:22 smithi115 ceph-mon[108015]: pgmap v273: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:23.041 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:48:23.270 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:48:23.747 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:23 smithi115 ceph-mon[108015]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:48:24.386 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:48:24.386 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:48:24.688 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:24 smithi115 ceph-mon[108015]: pgmap v274: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:24.798 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:48:25.799 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:48:25.832 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:25 smithi115 ceph-mon[108015]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:48:26.030 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:48:26.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:26 smithi115 ceph-mon[108015]: pgmap v275: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:27.139 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:48:27.140 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:48:27.548 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:48:28.549 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:48:28.781 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:48:28.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:28 smithi115 ceph-mon[108015]: pgmap v276: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:28.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:28 smithi115 ceph-mon[108015]: from='client.14468 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:48:29.919 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:48:29.919 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:48:30.275 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:48:30.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:30 smithi115 ceph-mon[108015]: pgmap v277: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:31.276 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:48:31.505 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:48:31.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:31 smithi115 ceph-mon[108015]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:48:32.640 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:48:32.640 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:48:32.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:32 smithi115 ceph-mon[108015]: pgmap v278: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:33.060 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:48:34.061 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:48:34.291 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:48:34.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:34 smithi115 ceph-mon[108015]: pgmap v279: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:34.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:34 smithi115 ceph-mon[108015]: from='client.14472 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:48:35.376 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:48:35.377 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:48:35.762 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:48:36.764 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:48:36.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:36 smithi115 ceph-mon[108015]: pgmap v280: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:36.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:36 smithi115 ceph-mon[108015]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:48:36.994 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:48:38.171 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:48:38.171 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:48:38.537 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:48:38.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:38 smithi115 ceph-mon[108015]: pgmap v281: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:39.538 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:48:39.770 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:48:39.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:39 smithi115 ceph-mon[108015]: from='client.14476 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:48:40.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:40 smithi115 ceph-mon[108015]: pgmap v282: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:40.940 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:48:40.940 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:48:41.319 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:48:41.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:41 smithi115 ceph-mon[108015]: pgmap v283: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:41.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:41 smithi115 ceph-mon[108015]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:48:42.319 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:48:42.549 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:48:43.756 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:48:43.756 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:48:44.135 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:48:44.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:44 smithi115 ceph-mon[108015]: pgmap v284: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:45.136 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:48:45.366 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:48:45.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:45 smithi115 ceph-mon[108015]: from='client.14480 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:48:46.526 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:48:46.527 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:48:46.738 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:46 smithi115 ceph-mon[108015]: pgmap v285: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:46.910 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:48:47.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:47 smithi115 ceph-mon[108015]: from='client.14482 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:48:47.910 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:48:48.142 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:48:48.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:48 smithi115 ceph-mon[108015]: pgmap v286: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:49.288 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:48:49.288 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:48:49.626 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:48:49.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:49 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:48:49.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:49 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:48:49.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:49 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:48:49.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:49 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:48:50.627 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:48:50.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:50 smithi115 ceph-mon[108015]: pgmap v287: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:50.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:50 smithi115 ceph-mon[108015]: pgmap v288: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:50.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:50 smithi115 ceph-mon[108015]: pgmap v289: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:50.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:50 smithi115 ceph-mon[108015]: from='client.14484 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:48:50.858 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:48:51.984 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:48:51.984 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:48:52.367 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:48:52.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:52 smithi115 ceph-mon[108015]: pgmap v290: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:53.367 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:48:53.597 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:48:53.709 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:53 smithi115 ceph-mon[108015]: from='client.14486 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:48:54.777 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:48:54.777 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:48:54.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:54 smithi115 ceph-mon[108015]: pgmap v291: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:55.175 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:48:56.176 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:48:56.406 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:48:56.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:56 smithi115 ceph-mon[108015]: from='client.14488 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:48:56.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:56 smithi115 ceph-mon[108015]: pgmap v292: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:57.605 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:48:57.605 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:48:58.008 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:48:58.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:58 smithi115 ceph-mon[108015]: pgmap v293: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:48:59.009 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:48:59.241 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:48:59.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:48:59 smithi115 ceph-mon[108015]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:49:00.350 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:49:00.350 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:49:00.741 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:49:00.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:00 smithi115 ceph-mon[108015]: pgmap v294: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:01.742 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:49:01.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:01 smithi115 ceph-mon[108015]: from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:49:01.974 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:49:02.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:02 smithi115 ceph-mon[108015]: pgmap v295: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:03.102 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:49:03.102 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:49:03.491 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:49:04.492 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:49:04.725 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:49:04.831 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:04 smithi115 ceph-mon[108015]: pgmap v296: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:04.832 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:04 smithi115 ceph-mon[108015]: from='client.14494 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:49:05.867 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:49:05.867 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:49:06.247 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:49:06.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:06 smithi115 ceph-mon[108015]: pgmap v297: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:07.247 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:49:07.476 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:49:07.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:07 smithi115 ceph-mon[108015]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:49:08.628 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:49:08.628 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:49:08.816 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:08 smithi115 ceph-mon[108015]: pgmap v298: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:09.013 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:49:10.014 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:49:10.243 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:49:10.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:10 smithi115 ceph-mon[108015]: from='client.14498 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:49:10.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:10 smithi115 ceph-mon[108015]: pgmap v299: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:11.423 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:49:11.423 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:49:11.811 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:49:12.812 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:49:12.846 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:12 smithi115 ceph-mon[108015]: pgmap v300: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:12.846 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:12 smithi115 ceph-mon[108015]: from='client.14500 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:49:13.047 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:49:14.181 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:49:14.181 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:49:14.614 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:49:14.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:14 smithi115 ceph-mon[108015]: pgmap v301: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:15.614 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:49:15.845 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:49:15.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:15 smithi115 ceph-mon[108015]: from='client.14502 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:49:16.622 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:16 smithi115 ceph-mon[108015]: pgmap v302: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:17.001 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:49:17.001 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:49:17.395 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:49:18.396 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:49:18.625 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:49:18.738 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:18 smithi115 ceph-mon[108015]: from='client.14504 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:49:18.738 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:18 smithi115 ceph-mon[108015]: pgmap v303: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:19.806 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:49:19.806 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:49:20.188 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:49:20.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:20 smithi115 ceph-mon[108015]: pgmap v304: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:21.189 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:49:21.421 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:49:21.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:21 smithi115 ceph-mon[108015]: from='client.14506 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:49:22.579 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:49:22.579 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:49:22.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:22 smithi115 ceph-mon[108015]: pgmap v305: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:22.952 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:49:23.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:23 smithi115 ceph-mon[108015]: from='client.14508 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:49:23.952 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:49:24.183 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:49:24.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:24 smithi115 ceph-mon[108015]: pgmap v306: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:25.345 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:49:25.346 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:49:25.777 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:49:25.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:25 smithi115 ceph-mon[108015]: pgmap v307: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:25.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:25 smithi115 ceph-mon[108015]: from='client.14510 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:49:26.778 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:49:27.008 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:49:28.110 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:49:28.111 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:49:28.645 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:49:28.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:28 smithi115 ceph-mon[108015]: pgmap v308: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:29.647 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:49:29.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:29 smithi115 ceph-mon[108015]: from='client.14512 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:49:29.878 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:49:30.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:30 smithi115 ceph-mon[108015]: pgmap v309: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:31.035 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:49:31.035 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:49:31.415 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:49:32.417 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:49:32.650 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:49:32.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:32 smithi115 ceph-mon[108015]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:49:32.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:32 smithi115 ceph-mon[108015]: pgmap v310: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:33.841 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:49:33.841 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:49:34.208 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:49:34.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:34 smithi115 ceph-mon[108015]: pgmap v311: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:35.208 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:49:35.439 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:49:35.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:35 smithi115 ceph-mon[108015]: from='client.14516 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:49:36.575 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:49:36.575 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:49:36.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:36 smithi115 ceph-mon[108015]: pgmap v312: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:36.956 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:49:37.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:37 smithi115 ceph-mon[108015]: from='client.14518 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:49:37.958 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:49:38.187 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:49:38.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:38 smithi115 ceph-mon[108015]: pgmap v313: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:39.307 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:49:39.307 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:49:39.698 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:49:40.699 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:49:40.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:40 smithi115 ceph-mon[108015]: pgmap v314: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:40.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:40 smithi115 ceph-mon[108015]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:49:40.929 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:49:42.070 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:49:42.070 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:49:42.440 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:49:42.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:42 smithi115 ceph-mon[108015]: pgmap v315: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:43.441 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:49:43.672 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:49:43.785 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:43 smithi115 ceph-mon[108015]: from='client.14522 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:49:44.838 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:49:44.839 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:49:44.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:44 smithi115 ceph-mon[108015]: pgmap v316: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:45.223 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:49:46.225 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:49:46.454 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:49:46.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:46 smithi115 ceph-mon[108015]: from='client.14524 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:49:46.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:46 smithi115 ceph-mon[108015]: pgmap v317: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:47.608 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:49:47.608 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:49:47.979 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:49:48.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:48 smithi115 ceph-mon[108015]: pgmap v318: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:48.981 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:49:49.215 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:49:49.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:49 smithi115 ceph-mon[108015]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:49:49.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:49 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:49:49.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:49 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:49:49.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:49 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:49:50.343 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:49:50.344 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:49:50.730 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:49:50.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:50 smithi115 ceph-mon[108015]: pgmap v319: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:50.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:50 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:49:50.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:50 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:49:50.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:50 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:49:51.731 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:49:51.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:51 smithi115 ceph-mon[108015]: pgmap v320: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:51.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:51 smithi115 ceph-mon[108015]: pgmap v321: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:51.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:51 smithi115 ceph-mon[108015]: from='client.14528 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:49:51.962 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:49:53.079 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:49:53.079 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:49:53.470 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:49:53.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:53 smithi115 ceph-mon[108015]: pgmap v322: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:54.470 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:49:54.728 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:49:54.838 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:54 smithi115 ceph-mon[108015]: from='client.14530 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:49:55.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:55 smithi115 ceph-mon[108015]: pgmap v323: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:55.863 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:49:55.864 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:49:56.254 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:49:57.254 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:49:57.484 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:49:57.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:57 smithi115 ceph-mon[108015]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:49:57.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:57 smithi115 ceph-mon[108015]: pgmap v324: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:49:58.649 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:49:58.649 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:49:59.027 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:49:59.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:49:59 smithi115 ceph-mon[108015]: pgmap v325: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:00.028 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:50:00.262 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:50:00.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:00 smithi115 ceph-mon[108015]: from='client.14534 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:50:01.388 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:50:01.388 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:50:01.787 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:50:01.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:01 smithi115 ceph-mon[108015]: pgmap v326: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:02.788 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:50:02.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:02 smithi115 ceph-mon[108015]: from='client.14536 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:50:03.020 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:50:03.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:03 smithi115 ceph-mon[108015]: pgmap v327: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:04.213 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:50:04.213 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:50:04.646 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:04 smithi115 ceph-mon[108015]: pgmap v328: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:04.647 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:04 smithi115 ceph-mon[108015]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:50:04.648 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:50:05.649 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:50:05.879 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:50:07.046 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:50:07.046 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:50:07.229 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:07 smithi115 ceph-mon[108015]: pgmap v329: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:07.417 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:50:08.418 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:50:08.645 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:50:08.793 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:08 smithi115 ceph-mon[108015]: from='client.14540 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:50:09.753 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:09 smithi115 ceph-mon[108015]: pgmap v330: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:09.816 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:50:09.816 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:50:10.200 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:50:11.202 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:50:11.432 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:50:11.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:11 smithi115 ceph-mon[108015]: from='client.14542 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:50:11.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:11 smithi115 ceph-mon[108015]: pgmap v331: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:12.605 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:50:12.605 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:50:12.983 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:50:13.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:13 smithi115 ceph-mon[108015]: pgmap v332: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:13.984 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:50:14.212 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:50:14.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:14 smithi115 ceph-mon[108015]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:50:15.384 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:50:15.384 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:50:15.767 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:50:15.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:15 smithi115 ceph-mon[108015]: pgmap v333: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:16.768 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:50:16.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:16 smithi115 ceph-mon[108015]: from='client.14546 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:50:17.000 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:50:17.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:17 smithi115 ceph-mon[108015]: pgmap v334: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:18.142 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:50:18.142 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:50:18.522 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:50:19.523 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:50:19.756 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:50:19.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:19 smithi115 ceph-mon[108015]: pgmap v335: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:19.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:19 smithi115 ceph-mon[108015]: from='client.14548 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:50:20.908 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:50:20.908 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:50:21.288 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:50:21.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:21 smithi115 ceph-mon[108015]: pgmap v336: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:22.290 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:50:22.519 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:50:22.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:22 smithi115 ceph-mon[108015]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:50:23.699 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:50:23.699 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:50:23.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:23 smithi115 ceph-mon[108015]: pgmap v337: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:24.071 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:50:25.072 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:50:25.303 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:50:25.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:25 smithi115 ceph-mon[108015]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:50:25.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:25 smithi115 ceph-mon[108015]: pgmap v338: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:26.410 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:50:26.410 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:50:26.787 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:50:27.788 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:50:27.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:27 smithi115 ceph-mon[108015]: pgmap v339: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:27.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:27 smithi115 ceph-mon[108015]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:50:28.021 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:50:29.168 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:50:29.168 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:50:29.561 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:50:29.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:29 smithi115 ceph-mon[108015]: pgmap v340: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:30.562 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:50:30.791 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:50:30.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:30 smithi115 ceph-mon[108015]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:50:31.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:31 smithi115 ceph-mon[108015]: pgmap v341: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:31.882 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:50:31.882 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:50:32.287 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:50:33.289 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:50:33.518 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:50:33.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:33 smithi115 ceph-mon[108015]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:50:33.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:33 smithi115 ceph-mon[108015]: pgmap v342: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:34.632 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:50:34.632 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:50:35.036 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:50:35.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:35 smithi115 ceph-mon[108015]: pgmap v343: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:36.037 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:50:36.268 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:50:36.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:36 smithi115 ceph-mon[108015]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:50:37.398 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:50:37.398 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:50:37.809 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:50:38.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:37 smithi115 ceph-mon[108015]: pgmap v344: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:38.809 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:50:39.041 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:50:39.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:38 smithi115 ceph-mon[108015]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:50:39.824 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:39 smithi115 ceph-mon[108015]: pgmap v345: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:40.191 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:50:40.191 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:50:40.744 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:50:40.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:40 smithi115 ceph-mon[108015]: pgmap v346: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:40.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:40 smithi115 ceph-mon[108015]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:50:41.746 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:50:41.974 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:50:43.090 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:50:43.091 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:50:43.493 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:50:43.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:43 smithi115 ceph-mon[108015]: pgmap v347: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:44.495 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:50:44.728 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:50:44.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:44 smithi115 ceph-mon[108015]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:50:45.825 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:45 smithi115 ceph-mon[108015]: pgmap v348: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:45.924 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:50:45.924 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:50:46.291 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:50:47.292 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:50:47.523 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:50:47.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:47 smithi115 ceph-mon[108015]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:50:47.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:47 smithi115 ceph-mon[108015]: pgmap v349: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:48.636 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:50:48.636 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:50:49.025 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:50:49.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:49 smithi115 ceph-mon[108015]: pgmap v350: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:50.025 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:50:50.270 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:50:50.822 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:50 smithi115 ceph-mon[108015]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:50:50.822 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:50 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:50:50.823 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:50 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-09T07:50:50.823 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:50 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-09T07:50:50.823 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:50 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:50:50.823 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:50 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:50:50.823 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:50 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:50:50.823 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:50 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:50:51.410 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:50:51.411 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:50:51.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:51 smithi115 ceph-mon[108015]: pgmap v351: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:51.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:51 smithi115 ceph-mon[108015]: Adjusting osd_memory_target on smithi115 to 8044M 2023-10-09T07:50:51.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:51 smithi115 ceph-mon[108015]: pgmap v352: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:51.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:51 smithi115 ceph-mon[108015]: pgmap v353: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:51.791 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:50:52.793 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:50:52.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:52 smithi115 ceph-mon[108015]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:50:53.022 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:50:53.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:53 smithi115 ceph-mon[108015]: pgmap v354: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:54.155 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:50:54.155 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:50:54.558 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:50:55.559 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:50:55.790 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:50:55.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:55 smithi115 ceph-mon[108015]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:50:55.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:55 smithi115 ceph-mon[108015]: pgmap v355: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:56.943 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:50:56.943 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:50:57.348 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:50:57.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:57 smithi115 ceph-mon[108015]: pgmap v356: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:50:58.348 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:50:58.578 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:50:58.698 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:58 smithi115 ceph-mon[108015]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:50:59.706 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:50:59.707 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:50:59.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:50:59 smithi115 ceph-mon[108015]: pgmap v357: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:00.090 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:51:01.091 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:51:01.323 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:51:01.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:01 smithi115 ceph-mon[108015]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:51:01.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:01 smithi115 ceph-mon[108015]: pgmap v358: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:02.476 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:51:02.477 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:51:02.854 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:51:03.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:03 smithi115 ceph-mon[108015]: pgmap v359: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:03.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:03 smithi115 ceph-mon[108015]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:51:03.855 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:51:04.083 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:51:05.235 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:51:05.236 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:51:05.595 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:51:05.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:05 smithi115 ceph-mon[108015]: pgmap v360: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:06.595 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:51:06.824 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:51:06.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:06 smithi115 ceph-mon[108015]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:51:07.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:07 smithi115 ceph-mon[108015]: pgmap v361: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:08.038 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:51:08.038 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:51:08.429 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:51:09.430 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:51:09.661 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:51:09.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:09 smithi115 ceph-mon[108015]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:51:09.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:09 smithi115 ceph-mon[108015]: pgmap v362: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:10.747 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:51:10.748 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:51:11.176 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:51:12.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:11 smithi115 ceph-mon[108015]: pgmap v363: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:12.177 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:51:12.408 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:51:12.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:12 smithi115 ceph-mon[108015]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:51:13.522 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:51:13.522 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:51:13.739 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:13 smithi115 ceph-mon[108015]: pgmap v364: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:13.739 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:13 smithi115 ceph-mon[108015]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:51:13.934 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:51:14.935 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:51:14.967 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:14 smithi115 ceph-mon[108015]: pgmap v365: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:15.163 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:51:16.255 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:51:16.502 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:51:16.660 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:51:17.661 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:51:17.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:17 smithi115 ceph-mon[108015]: from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:51:17.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:17 smithi115 ceph-mon[108015]: pgmap v366: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:17.891 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:51:18.985 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:51:18.985 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:51:19.334 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:51:19.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:19 smithi115 ceph-mon[108015]: pgmap v367: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:20.334 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:51:20.564 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:51:20.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:20 smithi115 ceph-mon[108015]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:51:21.649 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:51:21.649 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:51:21.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:21 smithi115 ceph-mon[108015]: pgmap v368: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:22.077 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:51:23.078 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:51:23.307 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:51:23.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:23 smithi115 ceph-mon[108015]: from='client.14594 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:51:23.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:23 smithi115 ceph-mon[108015]: pgmap v369: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:24.523 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:51:24.523 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:51:24.917 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:51:25.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:25 smithi115 ceph-mon[108015]: pgmap v370: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:25.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:25 smithi115 ceph-mon[108015]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:51:25.919 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:51:26.147 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:51:27.277 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:51:27.277 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:51:27.648 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:51:27.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:27 smithi115 ceph-mon[108015]: pgmap v371: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:28.649 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:51:28.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:28 smithi115 ceph-mon[108015]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:51:29.250 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:51:29.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:29 smithi115 ceph-mon[108015]: pgmap v372: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:30.037 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:51:30.038 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:51:30.433 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:51:31.434 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:51:31.663 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:51:31.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:31 smithi115 ceph-mon[108015]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:51:31.807 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:31 smithi115 ceph-mon[108015]: pgmap v373: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:32.828 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:51:32.828 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:51:33.214 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:51:33.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:33 smithi115 ceph-mon[108015]: pgmap v374: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:34.215 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:51:34.445 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:51:34.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:34 smithi115 ceph-mon[108015]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:51:35.585 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:51:35.586 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:51:35.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:35 smithi115 ceph-mon[108015]: pgmap v375: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:35.957 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:51:36.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:36 smithi115 ceph-mon[108015]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:51:36.958 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:51:37.191 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:51:37.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:37 smithi115 ceph-mon[108015]: pgmap v376: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:38.353 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:51:38.353 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:51:38.738 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:51:39.739 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:51:39.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:39 smithi115 ceph-mon[108015]: from='client.14606 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:51:39.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:39 smithi115 ceph-mon[108015]: pgmap v377: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:39.969 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:51:41.149 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:51:41.149 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:51:41.539 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:51:41.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:41 smithi115 ceph-mon[108015]: pgmap v378: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:42.540 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:51:42.771 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:51:42.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:42 smithi115 ceph-mon[108015]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:51:43.937 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:51:43.937 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:51:43.963 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:43 smithi115 ceph-mon[108015]: pgmap v379: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:44.313 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:51:45.314 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:51:45.544 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:51:46.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:45 smithi115 ceph-mon[108015]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:51:46.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:45 smithi115 ceph-mon[108015]: pgmap v380: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:46.725 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:51:46.726 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:51:47.162 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:51:48.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:47 smithi115 ceph-mon[108015]: pgmap v381: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:48.163 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:51:48.395 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:51:48.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:48 smithi115 ceph-mon[108015]: from='client.14612 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:51:49.499 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:51:49.499 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:51:49.747 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:49 smithi115 ceph-mon[108015]: pgmap v382: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:49.747 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:49 smithi115 ceph-mon[108015]: from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:51:49.928 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:51:50.930 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:51:50.955 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:50 smithi115 ceph-mon[108015]: pgmap v383: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:50.955 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:50 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:51:51.144 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:51:51.899 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:51 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:51:51.899 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:51 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:51:51.899 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:51 smithi115 ceph-mon[108015]: pgmap v384: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:51.899 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:51 smithi115 ceph-mon[108015]: pgmap v385: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:51.899 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:51 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:51:52.256 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:51:52.256 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:51:52.618 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:51:53.620 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:51:53.850 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:51:53.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:53 smithi115 ceph-mon[108015]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:51:54.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:54 smithi115 ceph-mon[108015]: pgmap v386: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:55.051 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:51:55.051 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:51:55.475 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:51:56.477 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:51:56.708 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:51:56.827 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:56 smithi115 ceph-mon[108015]: pgmap v387: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:56.827 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:56 smithi115 ceph-mon[108015]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:51:57.848 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:51:57.848 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:51:58.285 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:51:58.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:58 smithi115 ceph-mon[108015]: pgmap v388: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:51:59.286 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:51:59.520 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:51:59.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:51:59 smithi115 ceph-mon[108015]: from='client.14620 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:52:00.715 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:52:00.715 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:52:00.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:00 smithi115 ceph-mon[108015]: pgmap v389: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:01.117 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:52:02.118 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:52:02.349 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:52:02.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:02 smithi115 ceph-mon[108015]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:52:02.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:02 smithi115 ceph-mon[108015]: pgmap v390: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:03.535 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:52:03.535 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:52:03.915 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:52:04.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:04 smithi115 ceph-mon[108015]: pgmap v391: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:04.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:04 smithi115 ceph-mon[108015]: from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:52:04.917 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:52:05.146 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:52:06.251 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:52:06.251 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:52:06.642 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:52:06.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:06 smithi115 ceph-mon[108015]: pgmap v392: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:07.643 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:52:07.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:07 smithi115 ceph-mon[108015]: from='client.14626 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:52:07.873 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:52:08.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:08 smithi115 ceph-mon[108015]: pgmap v393: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:09.023 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:52:09.024 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:52:09.430 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:52:10.432 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:52:10.665 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:52:10.782 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:10 smithi115 ceph-mon[108015]: pgmap v394: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:10.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:10 smithi115 ceph-mon[108015]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:52:11.803 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:52:11.804 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:52:12.239 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:52:13.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:12 smithi115 ceph-mon[108015]: pgmap v395: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:13.240 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:52:13.470 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:52:13.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:13 smithi115 ceph-mon[108015]: from='client.14630 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:52:14.606 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:52:14.606 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:52:14.817 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:14 smithi115 ceph-mon[108015]: pgmap v396: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:15.023 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:52:16.024 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:52:16.056 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:15 smithi115 ceph-mon[108015]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:52:16.252 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:52:16.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:16 smithi115 ceph-mon[108015]: pgmap v397: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:17.405 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:52:17.405 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:52:17.821 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:52:18.822 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:52:19.053 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:52:19.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:18 smithi115 ceph-mon[108015]: pgmap v398: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:19.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:18 smithi115 ceph-mon[108015]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:52:20.196 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:52:20.196 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:52:20.603 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:52:20.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:20 smithi115 ceph-mon[108015]: pgmap v399: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:21.603 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:52:21.834 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:52:21.946 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:21 smithi115 ceph-mon[108015]: from='client.14636 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:52:22.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:22 smithi115 ceph-mon[108015]: pgmap v400: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:22.972 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:52:22.972 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:52:23.378 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:52:24.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:23 smithi115 ceph-mon[108015]: pgmap v401: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:24.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:23 smithi115 ceph-mon[108015]: from='client.14638 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:52:24.380 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:52:24.610 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:52:25.765 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:52:25.765 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:52:26.169 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:52:26.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:25 smithi115 ceph-mon[108015]: pgmap v402: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:27.170 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:52:27.204 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:26 smithi115 ceph-mon[108015]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:52:27.400 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:52:28.563 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:52:28.563 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:52:28.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:28 smithi115 ceph-mon[108015]: pgmap v403: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:28.959 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:52:29.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:29 smithi115 ceph-mon[108015]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:52:29.960 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:52:30.191 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:52:30.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:30 smithi115 ceph-mon[108015]: pgmap v404: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:31.308 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:52:31.308 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:52:31.709 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:52:32.710 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:52:32.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:32 smithi115 ceph-mon[108015]: pgmap v405: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:32.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:32 smithi115 ceph-mon[108015]: from='client.14644 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:52:32.940 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:52:34.094 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:52:34.095 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:52:34.461 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:52:34.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:34 smithi115 ceph-mon[108015]: pgmap v406: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:35.462 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:52:35.692 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:52:35.801 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:35 smithi115 ceph-mon[108015]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:52:36.840 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:52:36.841 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:52:36.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:36 smithi115 ceph-mon[108015]: pgmap v407: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:37.251 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:52:38.252 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:52:38.482 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:52:38.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:38 smithi115 ceph-mon[108015]: from='client.14648 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:52:38.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:38 smithi115 ceph-mon[108015]: pgmap v408: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:39.623 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:52:39.623 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:52:40.001 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:52:40.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:40 smithi115 ceph-mon[108015]: pgmap v409: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:40.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:40 smithi115 ceph-mon[108015]: from='client.14650 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:52:41.003 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:52:41.234 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:52:42.387 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:52:42.387 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:52:42.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:42 smithi115 ceph-mon[108015]: pgmap v410: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:42.866 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:52:43.867 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:52:43.900 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:43 smithi115 ceph-mon[108015]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:52:44.096 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:52:44.859 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:44 smithi115 ceph-mon[108015]: pgmap v411: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:45.230 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:52:45.231 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:52:45.606 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:52:46.607 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:52:46.838 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:52:46.941 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:46 smithi115 ceph-mon[108015]: pgmap v412: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:46.941 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:46 smithi115 ceph-mon[108015]: from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:52:47.964 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:52:47.964 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:52:48.424 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:52:49.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:48 smithi115 ceph-mon[108015]: pgmap v413: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:49.425 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:52:49.659 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:52:49.781 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:49 smithi115 ceph-mon[108015]: from='client.14656 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:52:50.914 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:52:50.914 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:52:50.936 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:50 smithi115 ceph-mon[108015]: pgmap v414: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:51.324 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:52:52.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:51 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:52:52.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:51 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:52:52.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:51 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:52:52.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:51 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:52:52.325 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:52:52.554 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:52:53.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:52 smithi115 ceph-mon[108015]: pgmap v415: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:53.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:52 smithi115 ceph-mon[108015]: from='client.14658 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:52:53.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:52 smithi115 ceph-mon[108015]: pgmap v416: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:53.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:52 smithi115 ceph-mon[108015]: pgmap v417: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:53.739 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:52:53.739 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:52:54.120 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:52:55.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:54 smithi115 ceph-mon[108015]: pgmap v418: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:55.121 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:52:55.349 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:52:55.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:55 smithi115 ceph-mon[108015]: from='client.14660 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:52:56.507 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:52:56.508 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:52:56.828 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:56 smithi115 ceph-mon[108015]: pgmap v419: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:56.879 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:52:57.880 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:52:57.913 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:57 smithi115 ceph-mon[108015]: from='client.14662 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:52:58.112 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:52:58.886 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:58 smithi115 ceph-mon[108015]: pgmap v420: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:52:59.242 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:52:59.242 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:52:59.620 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:52:59.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:59 smithi115 ceph-mon[108015]: from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:52:59.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:52:59 smithi115 ceph-mon[108015]: pgmap v421: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:00.622 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:53:00.855 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:53:01.992 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:53:01.992 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:53:02.369 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:53:02.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:02 smithi115 ceph-mon[108015]: pgmap v422: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:03.370 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:53:03.603 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:53:03.721 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:03 smithi115 ceph-mon[108015]: from='client.14666 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:53:04.735 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:53:04.736 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:53:04.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:04 smithi115 ceph-mon[108015]: pgmap v423: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:05.110 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:53:06.111 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:53:06.341 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:53:06.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:06 smithi115 ceph-mon[108015]: from='client.14668 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:53:06.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:06 smithi115 ceph-mon[108015]: pgmap v424: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:07.479 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:53:07.480 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:53:07.903 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:53:08.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:08 smithi115 ceph-mon[108015]: pgmap v425: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:08.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:08 smithi115 ceph-mon[108015]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:53:08.905 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:53:09.134 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:53:10.292 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:53:10.292 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:53:10.725 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:53:10.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:10 smithi115 ceph-mon[108015]: pgmap v426: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:11.727 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:53:11.959 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:53:12.086 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:11 smithi115 ceph-mon[108015]: from='client.14672 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:53:12.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:12 smithi115 ceph-mon[108015]: pgmap v427: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:13.124 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:53:13.124 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:53:13.534 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:53:14.535 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:53:14.767 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:53:14.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:14 smithi115 ceph-mon[108015]: from='client.14674 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:53:14.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:14 smithi115 ceph-mon[108015]: pgmap v428: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:15.897 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:53:15.897 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:53:16.332 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:53:17.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:16 smithi115 ceph-mon[108015]: pgmap v429: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:17.333 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:53:17.562 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:53:17.676 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:17 smithi115 ceph-mon[108015]: from='client.14676 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:53:18.724 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:53:18.724 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:53:18.919 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:18 smithi115 ceph-mon[108015]: pgmap v430: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:19.115 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:53:20.117 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:53:20.347 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:53:20.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:20 smithi115 ceph-mon[108015]: from='client.14678 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:53:20.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:20 smithi115 ceph-mon[108015]: pgmap v431: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:21.465 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:53:21.465 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:53:21.933 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:53:22.934 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:53:22.969 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:22 smithi115 ceph-mon[108015]: pgmap v432: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:22.969 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:22 smithi115 ceph-mon[108015]: from='client.14680 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:53:23.165 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:53:24.293 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:53:24.293 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:53:24.671 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:53:24.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:24 smithi115 ceph-mon[108015]: pgmap v433: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:25.672 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:53:25.902 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:53:26.041 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:25 smithi115 ceph-mon[108015]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:53:26.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:26 smithi115 ceph-mon[108015]: pgmap v434: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:27.066 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:53:27.066 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:53:27.515 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:53:28.516 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:53:28.746 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:53:28.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:28 smithi115 ceph-mon[108015]: from='client.14684 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:53:28.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:28 smithi115 ceph-mon[108015]: pgmap v435: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:29.928 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:53:29.928 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:53:30.356 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:53:31.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:30 smithi115 ceph-mon[108015]: pgmap v436: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:31.357 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:53:31.589 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:53:31.706 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:31 smithi115 ceph-mon[108015]: from='client.14686 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:53:32.736 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:53:32.737 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:53:32.907 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:32 smithi115 ceph-mon[108015]: pgmap v437: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:33.121 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:53:34.122 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:53:34.353 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:53:34.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:34 smithi115 ceph-mon[108015]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:53:34.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:34 smithi115 ceph-mon[108015]: pgmap v438: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:35.515 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:53:35.516 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:53:35.895 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:53:36.896 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:53:36.930 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:36 smithi115 ceph-mon[108015]: pgmap v439: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:36.930 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:36 smithi115 ceph-mon[108015]: from='client.14690 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:53:37.125 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:53:38.271 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:53:38.271 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:53:38.671 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:53:39.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:38 smithi115 ceph-mon[108015]: pgmap v440: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:39.672 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:53:39.902 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:53:40.005 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:39 smithi115 ceph-mon[108015]: from='client.14692 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:53:40.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:40 smithi115 ceph-mon[108015]: pgmap v441: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:41.023 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:53:41.023 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:53:41.394 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:53:42.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:41 smithi115 ceph-mon[108015]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:53:42.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:41 smithi115 ceph-mon[108015]: pgmap v442: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:42.395 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:53:42.626 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:53:43.805 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:53:43.805 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:53:44.202 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:53:44.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:44 smithi115 ceph-mon[108015]: pgmap v443: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:45.203 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:53:45.428 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:53:45.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:45 smithi115 ceph-mon[108015]: from='client.14696 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:53:46.579 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:53:46.579 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:53:46.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:46 smithi115 ceph-mon[108015]: pgmap v444: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:46.959 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:53:47.961 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:53:47.994 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:47 smithi115 ceph-mon[108015]: from='client.14698 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:53:48.191 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:53:48.930 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:48 smithi115 ceph-mon[108015]: pgmap v445: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:49.343 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:53:49.343 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:53:49.734 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:53:50.736 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:53:50.968 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:53:51.076 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:50 smithi115 ceph-mon[108015]: from='client.14700 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:53:51.077 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:50 smithi115 ceph-mon[108015]: pgmap v446: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:51.811 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:51 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:53:52.214 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:53:52.214 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:53:52.593 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:53:52.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:52 smithi115 ceph-mon[108015]: pgmap v447: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:53.594 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:53:53.812 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:53:53.967 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:53 smithi115 ceph-mon[108015]: from='client.14702 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:53:53.967 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:53 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:53:53.967 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:53 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:53:53.968 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:53 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:53:53.968 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:53 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:53:53.968 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:53 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:53:54.638 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:54 smithi115 ceph-mon[108015]: pgmap v448: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:54.638 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:54 smithi115 ceph-mon[108015]: pgmap v449: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:54.982 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:53:54.982 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:53:55.391 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:53:56.392 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:53:56.624 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:53:56.766 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:56 smithi115 ceph-mon[108015]: from='client.14704 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:53:56.767 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:56 smithi115 ceph-mon[108015]: pgmap v450: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:57.795 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:53:57.795 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:53:58.174 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:53:59.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:58 smithi115 ceph-mon[108015]: pgmap v451: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:53:59.175 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:53:59.406 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:53:59.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:53:59 smithi115 ceph-mon[108015]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:54:00.632 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:54:00.632 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:54:00.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:00 smithi115 ceph-mon[108015]: pgmap v452: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:01.041 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:54:02.042 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:54:02.075 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:01 smithi115 ceph-mon[108015]: from='client.14708 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:54:02.273 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:54:02.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:02 smithi115 ceph-mon[108015]: pgmap v453: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:03.451 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:54:03.452 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:54:03.823 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:54:04.824 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:54:05.055 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:54:05.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:04 smithi115 ceph-mon[108015]: pgmap v454: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:05.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:04 smithi115 ceph-mon[108015]: from='client.14710 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:54:06.264 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:54:06.265 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:54:06.655 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:54:06.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:06 smithi115 ceph-mon[108015]: pgmap v455: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:07.657 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:54:07.889 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:54:08.005 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:07 smithi115 ceph-mon[108015]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:54:08.976 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:54:08.976 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:54:08.978 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:08 smithi115 ceph-mon[108015]: pgmap v456: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:09.348 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:54:10.349 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:54:10.577 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:54:10.680 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:10 smithi115 ceph-mon[108015]: from='client.14714 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:54:10.680 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:10 smithi115 ceph-mon[108015]: pgmap v457: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:11.687 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:54:11.688 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:54:12.072 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:54:13.073 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:54:13.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:12 smithi115 ceph-mon[108015]: pgmap v458: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:13.304 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:54:13.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:13 smithi115 ceph-mon[108015]: from='client.14716 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:54:14.464 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:54:14.464 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:54:14.861 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:54:15.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:14 smithi115 ceph-mon[108015]: pgmap v459: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:15.862 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:54:15.895 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:15 smithi115 ceph-mon[108015]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:54:16.095 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:54:16.900 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:16 smithi115 ceph-mon[108015]: pgmap v460: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:17.253 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:54:17.253 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:54:17.649 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:54:18.651 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:54:18.880 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:54:19.011 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:18 smithi115 ceph-mon[108015]: pgmap v461: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:19.012 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:18 smithi115 ceph-mon[108015]: from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:54:20.080 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:54:20.080 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:54:20.462 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:54:21.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:20 smithi115 ceph-mon[108015]: pgmap v462: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:21.462 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:54:21.692 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:54:21.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:21 smithi115 ceph-mon[108015]: from='client.14722 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:54:22.920 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:54:23.296 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:54:23.297 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:22 smithi115 ceph-mon[108015]: pgmap v463: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:23.339 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:54:24.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:23 smithi115 ceph-mon[108015]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:54:24.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:23 smithi115 ceph-mon[108015]: pgmap v464: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:24.449 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:54:24.678 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:54:25.852 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:54:25.852 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:54:26.250 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:54:26.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:26 smithi115 ceph-mon[108015]: pgmap v465: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:27.251 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:54:27.482 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:54:27.593 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:27 smithi115 ceph-mon[108015]: from='client.14726 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:54:28.639 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:54:28.639 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:54:28.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:28 smithi115 ceph-mon[108015]: pgmap v466: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:29.035 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:54:30.036 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:54:30.069 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:29 smithi115 ceph-mon[108015]: from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:54:30.264 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:54:30.831 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:30 smithi115 ceph-mon[108015]: pgmap v467: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:31.395 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:54:31.395 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:54:31.792 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:54:32.792 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:54:33.022 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:54:33.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:32 smithi115 ceph-mon[108015]: pgmap v468: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:33.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:32 smithi115 ceph-mon[108015]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:54:34.157 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:54:34.158 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:54:34.549 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:54:34.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:34 smithi115 ceph-mon[108015]: pgmap v469: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:35.550 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:54:35.783 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:54:35.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:35 smithi115 ceph-mon[108015]: from='client.14732 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:54:36.926 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:54:36.927 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:54:36.956 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:36 smithi115 ceph-mon[108015]: pgmap v470: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:37.296 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:54:38.297 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:54:38.523 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:54:39.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:38 smithi115 ceph-mon[108015]: from='client.14734 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:54:39.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:38 smithi115 ceph-mon[108015]: pgmap v471: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:39.677 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:54:39.677 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:54:40.073 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:54:41.074 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:54:41.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:40 smithi115 ceph-mon[108015]: pgmap v472: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:41.304 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:54:41.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:41 smithi115 ceph-mon[108015]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:54:42.513 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:54:42.514 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:54:42.715 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:42 smithi115 ceph-mon[108015]: pgmap v473: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:42.906 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:54:43.907 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:54:43.940 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:43 smithi115 ceph-mon[108015]: from='client.14738 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:54:44.138 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:54:44.958 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:44 smithi115 ceph-mon[108015]: pgmap v474: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:45.334 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:54:45.335 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:54:45.762 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:54:46.763 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:54:46.995 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:54:47.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:46 smithi115 ceph-mon[108015]: pgmap v475: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:47.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:46 smithi115 ceph-mon[108015]: from='client.14740 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:54:48.155 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:54:48.155 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:54:48.582 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:54:48.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:48 smithi115 ceph-mon[108015]: pgmap v476: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:49.582 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:54:49.813 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:54:49.929 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:49 smithi115 ceph-mon[108015]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:54:50.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:50 smithi115 ceph-mon[108015]: pgmap v477: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:50.966 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:54:50.967 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:54:51.430 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:54:52.432 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:54:52.661 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:54:52.779 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:52 smithi115 ceph-mon[108015]: from='client.14744 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:54:52.779 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:52 smithi115 ceph-mon[108015]: pgmap v478: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:53.878 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:54:53.879 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:54:53.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:53 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:54:54.280 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:54:55.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:54 smithi115 ceph-mon[108015]: pgmap v479: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:55.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:54 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:54:55.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:54 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:54:55.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:54 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:54:55.281 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:54:55.512 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:54:56.084 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:55 smithi115 ceph-mon[108015]: pgmap v480: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:56.084 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:55 smithi115 ceph-mon[108015]: pgmap v481: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:56.084 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:55 smithi115 ceph-mon[108015]: from='client.14746 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:54:56.662 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:54:56.662 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:54:57.051 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:54:58.052 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:54:58.085 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:57 smithi115 ceph-mon[108015]: pgmap v482: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:54:58.284 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:54:58.988 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:58 smithi115 ceph-mon[108015]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:54:59.405 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:54:59.405 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:54:59.794 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:55:00.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:54:59 smithi115 ceph-mon[108015]: pgmap v483: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:00.795 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:55:01.026 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:55:01.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:00 smithi115 ceph-mon[108015]: from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:55:02.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:01 smithi115 ceph-mon[108015]: pgmap v484: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:02.246 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:55:02.246 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:55:02.642 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:55:03.642 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:55:03.871 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:55:04.018 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:03 smithi115 ceph-mon[108015]: pgmap v485: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:04.018 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:03 smithi115 ceph-mon[108015]: from='client.14752 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:55:05.058 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:55:05.059 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:55:05.433 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:55:06.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:05 smithi115 ceph-mon[108015]: pgmap v486: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:06.434 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:55:06.665 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:55:06.809 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:06 smithi115 ceph-mon[108015]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:55:07.815 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:55:07.815 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:55:08.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:07 smithi115 ceph-mon[108015]: pgmap v487: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:08.200 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:55:09.201 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:55:09.432 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:55:09.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:09 smithi115 ceph-mon[108015]: pgmap v488: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:09.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:09 smithi115 ceph-mon[108015]: from='client.14756 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:55:10.596 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:55:10.597 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:55:10.981 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:55:11.982 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:55:12.015 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:11 smithi115 ceph-mon[108015]: pgmap v489: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:12.015 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:11 smithi115 ceph-mon[108015]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:55:12.212 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:55:13.378 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:55:13.378 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:55:13.773 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:55:13.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:13 smithi115 ceph-mon[108015]: pgmap v490: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:14.774 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:55:15.003 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:55:15.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:14 smithi115 ceph-mon[108015]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:55:15.860 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:15 smithi115 ceph-mon[108015]: pgmap v491: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:16.219 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:55:16.219 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:55:16.622 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:55:17.623 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:55:17.989 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:17 smithi115 ceph-mon[108015]: pgmap v492: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:17.989 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:17 smithi115 ceph-mon[108015]: from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:55:18.185 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:55:19.318 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:55:19.318 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:55:19.673 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:55:19.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:19 smithi115 ceph-mon[108015]: pgmap v493: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:20.674 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:55:20.904 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:55:21.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:20 smithi115 ceph-mon[108015]: from='client.14764 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:55:21.745 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:21 smithi115 ceph-mon[108015]: pgmap v494: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:22.097 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:55:22.098 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:55:22.521 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:55:23.522 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:55:23.755 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:55:23.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:23 smithi115 ceph-mon[108015]: pgmap v495: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:23.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:23 smithi115 ceph-mon[108015]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:55:24.899 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:55:24.899 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:55:25.321 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:55:26.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:25 smithi115 ceph-mon[108015]: pgmap v496: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:26.322 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:55:26.553 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:55:27.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:26 smithi115 ceph-mon[108015]: from='client.14768 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:55:27.744 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:55:27.744 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:55:27.957 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:27 smithi115 ceph-mon[108015]: pgmap v497: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:28.169 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:55:29.170 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:55:29.402 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:55:29.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:29 smithi115 ceph-mon[108015]: pgmap v498: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:29.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:29 smithi115 ceph-mon[108015]: from='client.14770 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:55:30.559 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:55:30.559 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:55:30.977 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:55:31.978 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:55:32.011 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:31 smithi115 ceph-mon[108015]: pgmap v499: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:32.011 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:31 smithi115 ceph-mon[108015]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:55:32.208 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:55:33.356 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:55:33.356 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:55:33.735 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:55:34.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:33 smithi115 ceph-mon[108015]: pgmap v500: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:34.736 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:55:34.965 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:55:35.087 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:34 smithi115 ceph-mon[108015]: from='client.14774 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:55:35.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:35 smithi115 ceph-mon[108015]: pgmap v501: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:36.109 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:55:36.109 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:55:36.484 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:55:37.486 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:55:37.721 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:55:37.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:37 smithi115 ceph-mon[108015]: pgmap v502: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:37.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:37 smithi115 ceph-mon[108015]: from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:55:38.863 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:55:38.863 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:55:39.258 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:55:40.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:39 smithi115 ceph-mon[108015]: pgmap v503: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:40.259 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:55:40.490 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:55:40.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:40 smithi115 ceph-mon[108015]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:55:41.699 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:55:41.699 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:55:41.828 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:41 smithi115 ceph-mon[108015]: pgmap v504: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:42.084 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:55:43.085 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:55:43.310 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:55:43.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:43 smithi115 ceph-mon[108015]: from='client.14780 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:55:43.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:43 smithi115 ceph-mon[108015]: pgmap v505: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:44.459 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:55:44.459 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:55:44.840 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:55:45.841 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:55:46.071 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:55:46.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:45 smithi115 ceph-mon[108015]: pgmap v506: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:46.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:45 smithi115 ceph-mon[108015]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:55:47.245 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:55:47.245 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:55:47.641 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:55:47.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:47 smithi115 ceph-mon[108015]: pgmap v507: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:48.641 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:55:48.873 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:55:48.983 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:48 smithi115 ceph-mon[108015]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:55:49.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:49 smithi115 ceph-mon[108015]: pgmap v508: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:50.088 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:55:50.088 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:55:50.464 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:55:51.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:50 smithi115 ceph-mon[108015]: pgmap v509: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:51.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:50 smithi115 ceph-mon[108015]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:55:51.465 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:55:51.693 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:55:52.884 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:55:52.884 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:55:53.272 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:55:54.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:53 smithi115 ceph-mon[108015]: pgmap v510: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:54.274 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:55:54.488 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:55:55.054 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:54 smithi115 ceph-mon[108015]: from='client.14788 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:55:55.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:54 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:55:55.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:54 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:55:55.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:54 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:55:55.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:54 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:55:55.677 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:55:55.677 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:55:55.885 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:55 smithi115 ceph-mon[108015]: pgmap v511: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:55.885 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:55 smithi115 ceph-mon[108015]: pgmap v512: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:55.885 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:55 smithi115 ceph-mon[108015]: pgmap v513: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:56.063 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:55:57.065 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:55:57.297 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:55:57.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:57 smithi115 ceph-mon[108015]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:55:57.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:57 smithi115 ceph-mon[108015]: pgmap v514: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:58.553 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:55:58.553 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:55:58.913 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:55:59.915 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:55:59.949 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:59 smithi115 ceph-mon[108015]: pgmap v515: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:55:59.949 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:55:59 smithi115 ceph-mon[108015]: from='client.14792 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:56:00.147 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:56:01.315 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:56:01.598 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:56:01.737 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:56:01.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:01 smithi115 ceph-mon[108015]: pgmap v516: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:02.760 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:56:02.988 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:56:03.098 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:02 smithi115 ceph-mon[108015]: from='client.14794 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:56:03.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:03 smithi115 ceph-mon[108015]: pgmap v517: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:04.130 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:56:04.130 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:56:04.538 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:56:05.539 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:56:05.770 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:56:05.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:05 smithi115 ceph-mon[108015]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:56:05.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:05 smithi115 ceph-mon[108015]: pgmap v518: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:06.890 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:56:06.891 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:56:07.312 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:56:08.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:07 smithi115 ceph-mon[108015]: pgmap v519: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:08.313 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:56:08.542 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:56:09.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:08 smithi115 ceph-mon[108015]: from='client.14798 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:56:09.761 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:56:09.762 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:56:09.891 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:09 smithi115 ceph-mon[108015]: pgmap v520: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:10.219 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:56:11.221 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:56:11.451 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:56:11.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:11 smithi115 ceph-mon[108015]: from='client.14800 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:56:11.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:11 smithi115 ceph-mon[108015]: pgmap v521: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:12.628 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:56:12.628 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:56:13.018 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:56:14.019 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:56:14.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:13 smithi115 ceph-mon[108015]: pgmap v522: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:14.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:13 smithi115 ceph-mon[108015]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:56:14.249 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:56:15.418 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:56:15.418 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:56:15.794 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:56:16.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:15 smithi115 ceph-mon[108015]: pgmap v523: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:16.795 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:56:17.024 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:56:17.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:16 smithi115 ceph-mon[108015]: from='client.14804 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:56:18.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:17 smithi115 ceph-mon[108015]: pgmap v524: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:18.208 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:56:18.208 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:56:18.610 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:56:19.612 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:56:19.844 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:56:19.984 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:19 smithi115 ceph-mon[108015]: pgmap v525: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:19.985 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:19 smithi115 ceph-mon[108015]: from='client.14806 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:56:20.993 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:56:20.994 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:56:21.391 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:56:22.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:21 smithi115 ceph-mon[108015]: pgmap v526: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:22.392 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:56:22.625 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:56:22.741 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:22 smithi115 ceph-mon[108015]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:56:23.764 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:56:23.764 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:56:23.994 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:23 smithi115 ceph-mon[108015]: pgmap v527: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:24.183 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:56:25.184 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:56:25.414 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:56:25.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:25 smithi115 ceph-mon[108015]: from='client.14810 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:56:25.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:25 smithi115 ceph-mon[108015]: pgmap v528: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:26.567 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:56:26.567 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:56:26.982 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:56:27.984 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:56:28.016 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:27 smithi115 ceph-mon[108015]: pgmap v529: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:28.016 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:27 smithi115 ceph-mon[108015]: from='client.14812 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:56:28.214 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:56:29.405 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:56:29.405 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:56:29.789 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:56:29.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:29 smithi115 ceph-mon[108015]: pgmap v530: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:30.790 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:56:31.023 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:56:31.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:30 smithi115 ceph-mon[108015]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:56:32.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:31 smithi115 ceph-mon[108015]: pgmap v531: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:32.178 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:56:32.178 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:56:32.597 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:56:33.598 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:56:33.831 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:56:33.939 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:33 smithi115 ceph-mon[108015]: pgmap v532: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:33.939 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:33 smithi115 ceph-mon[108015]: from='client.14816 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:56:34.982 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:56:34.982 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:56:35.396 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:56:36.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:35 smithi115 ceph-mon[108015]: pgmap v533: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:36.398 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:56:36.627 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:56:36.740 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:36 smithi115 ceph-mon[108015]: from='client.14818 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:56:37.767 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:56:37.767 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:56:38.024 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:37 smithi115 ceph-mon[108015]: pgmap v534: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:38.223 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:56:39.224 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:56:39.456 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:56:39.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:39 smithi115 ceph-mon[108015]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:56:39.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:39 smithi115 ceph-mon[108015]: pgmap v535: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:40.635 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:56:40.635 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:56:41.014 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:56:42.015 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:56:42.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:41 smithi115 ceph-mon[108015]: pgmap v536: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:42.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:41 smithi115 ceph-mon[108015]: from='client.14822 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:56:42.245 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:56:43.359 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:56:43.359 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:56:43.777 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:56:44.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:43 smithi115 ceph-mon[108015]: pgmap v537: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:44.778 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:56:45.007 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:56:45.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:44 smithi115 ceph-mon[108015]: from='client.14824 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:56:45.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:45 smithi115 ceph-mon[108015]: pgmap v538: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:46.161 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:56:46.161 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:56:46.578 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:56:47.580 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:56:47.811 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:56:47.928 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:47 smithi115 ceph-mon[108015]: pgmap v539: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:47.928 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:47 smithi115 ceph-mon[108015]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:56:48.903 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:56:48.903 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:56:49.403 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:56:50.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:49 smithi115 ceph-mon[108015]: pgmap v540: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:50.403 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:56:50.635 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:56:50.768 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:50 smithi115 ceph-mon[108015]: from='client.14828 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:56:51.796 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:56:51.796 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:56:51.921 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:51 smithi115 ceph-mon[108015]: pgmap v541: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:52.159 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:56:53.160 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:56:53.391 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:56:53.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:53 smithi115 ceph-mon[108015]: from='client.14830 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:56:53.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:53 smithi115 ceph-mon[108015]: pgmap v542: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:54.546 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:56:54.547 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:56:54.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:54 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-09T07:56:54.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:54 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-09T07:56:54.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:54 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-09T07:56:54.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:54 smithi115 ceph-mon[108015]: from='mgr.14152 172.21.15.115:0/3797812697' entity='mgr.x' 2023-10-09T07:56:54.942 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:56:55.943 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:56:55.975 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:55 smithi115 ceph-mon[108015]: pgmap v543: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:55.975 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:55 smithi115 ceph-mon[108015]: from='client.14832 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:56:55.975 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:55 smithi115 ceph-mon[108015]: pgmap v544: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:55.975 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:55 smithi115 ceph-mon[108015]: pgmap v545: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:56.171 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:56:56.933 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:56 smithi115 ceph-mon[108015]: pgmap v546: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:56:57.305 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:56:57.305 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:56:57.693 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:56:58.694 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:56:58.925 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:56:59.034 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:58 smithi115 ceph-mon[108015]: from='client.14834 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:56:59.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:56:59 smithi115 ceph-mon[108015]: pgmap v547: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:00.162 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:57:00.162 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:57:00.551 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:57:01.552 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:57:01.785 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:57:01.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:01 smithi115 ceph-mon[108015]: from='client.14836 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:57:01.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:01 smithi115 ceph-mon[108015]: pgmap v548: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:02.914 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:57:02.915 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:57:03.299 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:57:04.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:03 smithi115 ceph-mon[108015]: pgmap v549: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:04.300 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:57:04.531 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:57:05.088 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:04 smithi115 ceph-mon[108015]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:57:05.669 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:57:05.669 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:57:06.064 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:05 smithi115 ceph-mon[108015]: pgmap v550: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:06.082 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:57:07.083 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:57:07.314 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:57:07.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:07 smithi115 ceph-mon[108015]: from='client.14840 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:57:07.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:07 smithi115 ceph-mon[108015]: pgmap v551: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:08.511 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:57:08.512 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:57:08.888 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:57:09.890 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:57:09.922 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:09 smithi115 ceph-mon[108015]: from='client.14842 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:57:09.922 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:09 smithi115 ceph-mon[108015]: pgmap v552: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:10.118 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:57:11.304 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:57:11.304 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:57:11.689 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:57:11.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:11 smithi115 ceph-mon[108015]: pgmap v553: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:12.691 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:57:12.921 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:57:13.081 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:12 smithi115 ceph-mon[108015]: from='client.14844 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:57:13.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:13 smithi115 ceph-mon[108015]: pgmap v554: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:14.122 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:57:14.123 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:57:14.623 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:57:15.625 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:57:15.855 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:57:15.971 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:15 smithi115 ceph-mon[108015]: from='client.14846 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:57:15.971 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:15 smithi115 ceph-mon[108015]: pgmap v555: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:16.988 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:57:16.988 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:57:17.387 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:57:18.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:17 smithi115 ceph-mon[108015]: pgmap v556: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:18.388 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:57:18.618 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:57:18.730 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:18 smithi115 ceph-mon[108015]: from='client.14848 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:57:19.772 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:57:19.772 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:57:19.965 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:19 smithi115 ceph-mon[108015]: pgmap v557: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:20.189 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:57:21.190 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:57:21.420 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:57:21.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:21 smithi115 ceph-mon[108015]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:57:21.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:21 smithi115 ceph-mon[108015]: pgmap v558: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:22.593 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:57:22.593 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:57:22.986 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:57:23.988 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:57:24.021 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:23 smithi115 ceph-mon[108015]: pgmap v559: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:24.021 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:23 smithi115 ceph-mon[108015]: from='client.14852 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:57:24.220 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:57:25.400 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:57:25.400 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:57:25.786 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:57:25.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:25 smithi115 ceph-mon[108015]: pgmap v560: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:26.787 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:57:27.016 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:57:27.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:26 smithi115 ceph-mon[108015]: from='client.14854 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:57:27.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:27 smithi115 ceph-mon[108015]: pgmap v561: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:28.147 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:57:28.147 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:57:28.544 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:57:29.544 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:57:29.771 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:57:29.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:29 smithi115 ceph-mon[108015]: from='client.14856 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:57:29.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:29 smithi115 ceph-mon[108015]: pgmap v562: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:30.924 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:57:30.924 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:57:31.326 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:57:32.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:31 smithi115 ceph-mon[108015]: pgmap v563: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:32.328 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:57:32.561 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:57:33.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:32 smithi115 ceph-mon[108015]: from='client.14858 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:57:33.729 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:57:33.729 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:57:34.091 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:33 smithi115 ceph-mon[108015]: pgmap v564: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:34.115 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:57:35.117 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:57:35.343 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:57:35.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:35 smithi115 ceph-mon[108015]: from='client.14860 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:57:35.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:35 smithi115 ceph-mon[108015]: pgmap v565: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:36.477 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:57:36.477 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:57:36.893 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:57:37.894 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:57:37.927 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:37 smithi115 ceph-mon[108015]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:57:37.927 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:37 smithi115 ceph-mon[108015]: pgmap v566: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:38.123 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:57:39.291 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:57:39.291 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:57:39.700 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:57:39.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:39 smithi115 ceph-mon[108015]: pgmap v567: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:40.701 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:57:40.932 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:57:41.033 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:40 smithi115 ceph-mon[108015]: from='client.14864 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:57:41.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:41 smithi115 ceph-mon[108015]: pgmap v568: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:42.056 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:57:42.056 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:57:42.423 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:57:43.425 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:57:43.655 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:57:43.765 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:43 smithi115 ceph-mon[108015]: from='client.14866 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:57:43.765 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:43 smithi115 ceph-mon[108015]: pgmap v569: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:44.816 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:57:44.817 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:57:45.207 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:57:46.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:45 smithi115 ceph-mon[108015]: pgmap v570: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:46.208 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:57:46.435 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:57:46.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:46 smithi115 ceph-mon[108015]: from='client.14868 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:57:47.633 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:57:47.633 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:57:47.817 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:47 smithi115 ceph-mon[108015]: pgmap v571: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:48.022 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:57:49.024 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 -- ceph orch ls -f json 2023-10-09T07:57:49.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:48 smithi115 ceph-mon[108015]: from='client.14870 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:57:49.249 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-09T07:57:49.991 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:49 smithi115 ceph-mon[108015]: pgmap v572: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:50.393 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-10-09T07:57:50.393 INFO:teuthology.orchestra.run.smithi115.stdout:[{"events": ["2023-10-09T07:40:10.992465Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-10-09T07:40:10.989934Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:40:05.411347Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115:172.21.15.115=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-10-09T07:40:05.408889Z", "running": 1, "size": 1}}, {"events": ["2023-10-09T07:43:34.243951Z service:nvmeof.mypool [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi115"]}, "service_id": "mypool", "service_name": "nvmeof.mypool", "service_type": "nvmeof", "spec": {"client_cert": "./client.crt", "client_key": "./client.key", "conn_retries": 10, "pool": "mypool", "port": 5500, "server_cert": "./server.crt", "server_key": "./server.key", "spdk_path": "/usr/local/bin/nvmf_tgt", "tgt_path": "/usr/local/bin/nvmf_tgt", "timeout": 60, "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp"}, "status": {"created": "2023-10-09T07:43:34.241216Z", "ports": [5500, 4420, 8009], "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7", "running": 2, "size": 2}, "unmanaged": true}] 2023-10-09T07:57:50.847 INFO:tasks.cephadm:nvmeof.mypool has 0/1 2023-10-09T07:57:50.848 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_barakda_ceph_6baafb588e77fa3b88e89ecedcf1f86c473eacbf/qa/tasks/cephadm.py", line 1374, in wait_for_service while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (301) after waiting for 300 seconds 2023-10-09T07:57:50.979 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=11a5ff5202dc4adeb7e76aa2ba8a8f12 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_barakda_ceph_6baafb588e77fa3b88e89ecedcf1f86c473eacbf/qa/tasks/cephadm.py", line 1374, in wait_for_service while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (301) after waiting for 300 seconds 2023-10-09T07:57:50.980 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-10-09T07:57:50.989 INFO:tasks.cephadm:Teardown begin 2023-10-09T07:57:50.989 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-10-09T07:57:51.029 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:50 smithi115 ceph-mon[108015]: from='client.14872 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-09T07:57:51.029 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:50 smithi115 ceph-mon[108015]: pgmap v573: 32 pgs: 5 active+undersized+degraded, 27 active+undersized; 581 B data, 53 MiB used, 179 GiB / 179 GiB avail; 5/15 objects degraded (33.333%) 2023-10-09T07:57:51.033 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-10-09T07:57:51.033 DEBUG:teuthology.orchestra.run.smithi115:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-10-09T07:57:51.091 INFO:tasks.cephadm:Stopping all daemons... 2023-10-09T07:57:51.092 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-10-09T07:57:51.092 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@mon.a 2023-10-09T07:57:51.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:51 smithi115 systemd[1]: Stopping Ceph mon.a for d56ccb5c-6676-11ee-8db5-212e2dc638e7... 2023-10-09T07:57:51.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:51 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mon-a[108011]: 2023-10-09T07:57:51.346+0000 7fa36174f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-10-09T07:57:51.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:51 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mon-a[108011]: 2023-10-09T07:57:51.346+0000 7fa36174f700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-10-09T07:57:52.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Oct 09 07:57:51 smithi115 bash[186098]: ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mon-a 2023-10-09T07:57:52.107 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@mon.a.service' 2023-10-09T07:57:52.155 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-09T07:57:52.156 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-10-09T07:57:52.156 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-10-09T07:57:52.156 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@mgr.x 2023-10-09T07:57:52.377 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:57:52 smithi115 systemd[1]: Stopping Ceph mgr.x for d56ccb5c-6676-11ee-8db5-212e2dc638e7... 2023-10-09T07:57:52.784 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:57:52 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:57:52.375+0000 7fae18b6b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-09T07:57:52.785 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:57:52 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x[108241]: 2023-10-09T07:57:52.375+0000 7fae18b6b700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-10-09T07:57:52.785 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Oct 09 07:57:52 smithi115 bash[186234]: ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-mgr-x 2023-10-09T07:57:52.789 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@mgr.x.service' 2023-10-09T07:57:52.824 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-09T07:57:52.824 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-10-09T07:57:52.825 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-10-09T07:57:52.825 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@osd.0 2023-10-09T07:57:53.070 INFO:journalctl@ceph.osd.0.smithi115.stdout:Oct 09 07:57:52 smithi115 systemd[1]: Stopping Ceph osd.0 for d56ccb5c-6676-11ee-8db5-212e2dc638e7... 2023-10-09T07:57:53.352 INFO:journalctl@ceph.osd.0.smithi115.stdout:Oct 09 07:57:53 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-osd-0[118766]: 2023-10-09T07:57:53.069+0000 7f4b88e04700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-09T07:57:53.352 INFO:journalctl@ceph.osd.0.smithi115.stdout:Oct 09 07:57:53 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-osd-0[118766]: 2023-10-09T07:57:53.069+0000 7f4b88e04700 -1 osd.0 19 *** Got signal Terminated *** 2023-10-09T07:57:53.352 INFO:journalctl@ceph.osd.0.smithi115.stdout:Oct 09 07:57:53 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-osd-0[118766]: 2023-10-09T07:57:53.069+0000 7f4b88e04700 -1 osd.0 19 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-09T07:57:58.852 INFO:journalctl@ceph.osd.0.smithi115.stdout:Oct 09 07:57:58 smithi115 bash[186370]: ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-osd-0 2023-10-09T07:57:59.689 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@osd.0.service' 2023-10-09T07:57:59.730 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-09T07:57:59.730 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-10-09T07:57:59.730 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-10-09T07:57:59.730 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@osd.1 2023-10-09T07:57:59.990 INFO:journalctl@ceph.osd.1.smithi115.stdout:Oct 09 07:57:59 smithi115 systemd[1]: Stopping Ceph osd.1 for d56ccb5c-6676-11ee-8db5-212e2dc638e7... 2023-10-09T07:58:00.352 INFO:journalctl@ceph.osd.1.smithi115.stdout:Oct 09 07:57:59 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-osd-1[122834]: 2023-10-09T07:57:59.988+0000 7fb0da52b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-09T07:58:00.352 INFO:journalctl@ceph.osd.1.smithi115.stdout:Oct 09 07:57:59 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-osd-1[122834]: 2023-10-09T07:57:59.988+0000 7fb0da52b700 -1 osd.1 19 *** Got signal Terminated *** 2023-10-09T07:58:00.352 INFO:journalctl@ceph.osd.1.smithi115.stdout:Oct 09 07:57:59 smithi115 ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-osd-1[122834]: 2023-10-09T07:57:59.988+0000 7fb0da52b700 -1 osd.1 19 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-09T07:58:05.852 INFO:journalctl@ceph.osd.1.smithi115.stdout:Oct 09 07:58:05 smithi115 bash[186611]: ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7-osd-1 2023-10-09T07:58:06.785 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d56ccb5c-6676-11ee-8db5-212e2dc638e7@osd.1.service' 2023-10-09T07:58:06.824 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-09T07:58:06.824 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-10-09T07:58:06.825 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 --force --keep-logs 2023-10-09T07:58:07.033 INFO:teuthology.orchestra.run.smithi115.stdout:Deleting cluster with fsid: d56ccb5c-6676-11ee-8db5-212e2dc638e7 2023-10-09T07:58:07.857 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-10-09T07:58:07.887 INFO:tasks.cephadm:Archiving crash dumps... 2023-10-09T07:58:07.889 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/crash to /home/teuthworker/archive/bdavidov-2023-10-09_07:20:03-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7418391/remote/smithi115/crash 2023-10-09T07:58:07.890 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/crash -- . 2023-10-09T07:58:07.965 INFO:teuthology.orchestra.run.smithi115.stderr:tar: /var/lib/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/crash: Cannot open: No such file or directory 2023-10-09T07:58:07.965 INFO:teuthology.orchestra.run.smithi115.stderr:tar: Error is not recoverable: exiting now 2023-10-09T07:58:07.966 INFO:tasks.cephadm:Checking cluster log for badness... 2023-10-09T07:58:07.966 DEBUG:teuthology.orchestra.run.smithi115:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2023-10-09T07:58:08.033 INFO:teuthology.orchestra.run.smithi115.stderr:grep: /var/log/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/ceph.log: No such file or directory 2023-10-09T07:58:08.034 INFO:tasks.cephadm:Compressing logs... 2023-10-09T07:58:08.035 DEBUG:teuthology.orchestra.run.smithi115:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-09T07:58:08.102 INFO:teuthology.orchestra.run.smithi115.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log : No such file or directory 2023-10-09T07:58:08.102 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-10-09T07:58:08.102 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/ceph-volume.log 2023-10-09T07:58:08.103 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/ceph-osd.0.log 2023-10-09T07:58:08.103 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/ceph-mgr.x.log 2023-10-09T07:58:08.103 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/ceph-mon.a.log 2023-10-09T07:58:08.104 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/ceph-osd.0.log: /var/log/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/ceph-osd.1.log 2023-10-09T07:58:08.104 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/ceph-mon.a.log: /var/log/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/ceph-osd.1.log: 91.7% -- replaced with /var/log/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/ceph-volume.log.gz 2023-10-09T07:58:08.110 INFO:teuthology.orchestra.run.smithi115.stderr: 95.8% -- replaced with /var/log/ceph/cephadm.log.gz 2023-10-09T07:58:08.216 INFO:teuthology.orchestra.run.smithi115.stderr: 91.3% -- replaced with /var/log/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/ceph-mgr.x.log.gz 2023-10-09T07:58:08.304 INFO:teuthology.orchestra.run.smithi115.stderr: 94.3% -- replaced with /var/log/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/ceph-osd.0.log.gz 2023-10-09T07:58:08.348 INFO:teuthology.orchestra.run.smithi115.stderr: 94.4% -- replaced with /var/log/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/ceph-osd.1.log.gz 2023-10-09T07:58:08.436 INFO:teuthology.orchestra.run.smithi115.stderr: 89.8% -- replaced with /var/log/ceph/d56ccb5c-6676-11ee-8db5-212e2dc638e7/ceph-mon.a.log.gz 2023-10-09T07:58:08.437 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-10-09T07:58:08.438 INFO:teuthology.orchestra.run.smithi115.stderr:real 0m0.345s 2023-10-09T07:58:08.438 INFO:teuthology.orchestra.run.smithi115.stderr:user 0m0.869s 2023-10-09T07:58:08.438 INFO:teuthology.orchestra.run.smithi115.stderr:sys 0m0.053s 2023-10-09T07:58:08.438 INFO:tasks.cephadm:Archiving logs... 2023-10-09T07:58:08.438 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/log/ceph to /home/teuthworker/archive/bdavidov-2023-10-09_07:20:03-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7418391/remote/smithi115/log 2023-10-09T07:58:08.439 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-09T07:58:08.668 INFO:tasks.cephadm:Removing cluster... 2023-10-09T07:58:08.669 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d56ccb5c-6676-11ee-8db5-212e2dc638e7 --force 2023-10-09T07:58:08.849 INFO:teuthology.orchestra.run.smithi115.stdout:Deleting cluster with fsid: d56ccb5c-6676-11ee-8db5-212e2dc638e7 2023-10-09T07:58:08.990 INFO:tasks.cephadm:Removing cephadm ... 2023-10-09T07:58:08.991 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf /home/ubuntu/cephtest/cephadm 2023-10-09T07:58:09.009 INFO:tasks.cephadm:Teardown complete 2023-10-09T07:58:09.009 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-09T07:58:09.020 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-09T07:58:09.020 DEBUG:teuthology.orchestra.run.smithi115:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-09T07:58:09.065 INFO:teuthology.orchestra.run.smithi115.stderr:bash: ntpq: command not found 2023-10-09T07:58:09.069 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-09T07:58:09.069 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2023-10-09T07:58:09.069 INFO:teuthology.orchestra.run.smithi115.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 29 -818us[ -818us] +/- 85ms 2023-10-09T07:58:09.070 INFO:teuthology.orchestra.run.smithi115.stdout:^* hv02.front.sepia.ceph.com 3 6 377 26 +195us[ +195us] +/- 67ms 2023-10-09T07:58:09.070 INFO:teuthology.orchestra.run.smithi115.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 +449us[ +449us] +/- 67ms 2023-10-09T07:58:09.070 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-10-09T07:58:09.071 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-09T07:58:09.079 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-09T07:58:09.080 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-09T07:58:09.088 DEBUG:teuthology.orchestra.run.smithi115:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-09T07:58:09.208 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-09T07:58:09.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-09T07:58:09.240 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-09T07:58:09.250 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-09T07:58:09.258 INFO:teuthology.task.internal:Duration was 1637.925376 seconds 2023-10-09T07:58:09.258 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-09T07:58:09.267 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-09T07:58:09.267 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-09T07:58:09.327 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-09T07:58:09.614 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-09T07:58:09.614 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi115.front.sepia.ceph.com 2023-10-09T07:58:09.615 DEBUG:teuthology.orchestra.run.smithi115:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-09T07:58:09.639 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-09T07:58:09.640 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-09T07:58:09.737 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-09T07:58:09.737 DEBUG:teuthology.orchestra.run.smithi115:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-09T07:58:09.850 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-09T07:58:09.861 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-09T07:58:09.861 DEBUG:teuthology.orchestra.run.smithi115:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-09T07:58:09.888 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-09T07:58:09.898 DEBUG:teuthology.orchestra.run.smithi115:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-09T07:58:09.954 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = core 2023-10-09T07:58:09.982 DEBUG:teuthology.orchestra.run.smithi115:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-09T07:58:10.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-09T07:58:10.042 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-09T07:58:10.052 INFO:teuthology.task.internal:Transferring archived files... 2023-10-09T07:58:10.053 DEBUG:teuthology.misc:Transferring archived files from smithi115:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/bdavidov-2023-10-09_07:20:03-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7418391/remote/smithi115 2023-10-09T07:58:10.053 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-09T07:58:10.125 INFO:teuthology.task.internal:Removing archive directory... 2023-10-09T07:58:10.125 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-09T07:58:10.175 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-09T07:58:10.188 INFO:teuthology.task.internal:Not uploading archives. 2023-10-09T07:58:10.188 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-09T07:58:10.197 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-09T07:58:10.198 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-09T07:58:10.232 INFO:teuthology.orchestra.run.smithi115.stdout: 525364 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 9 07:58 /home/ubuntu/cephtest 2023-10-09T07:58:10.234 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-09T07:58:10.244 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-09T07:58:10.264 INFO:teuthology.nuke:Checking targets against current locks 2023-10-09T07:58:10.283 DEBUG:teuthology.nuke:shortname: smithi115 2023-10-09T07:58:10.284 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-09T07:58:10.305 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/bdavidov-2023-10-09_07:20:03-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7418391', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-09 07:22:16.923236', 'locked_by': 'scheduled_bdavidov@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLzLt28sfKk++bxLIyz6AKjhC+jrWdfJvnbtygRh8hEIH11qIZgeVyMsC/Q+TKMwk01YkQ7vOdbRI0aiDX/tsNA='} 2023-10-09T07:58:10.349 INFO:teuthology.orchestra.console.smithi115:Power off 2023-10-09T07:58:10.350 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-09T07:58:10.372 DEBUG:teuthology.orchestra.console.smithi115:power off output: Chassis Power Control: Down/Off 2023-10-09T07:58:10.372 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-09T07:58:10.397 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is on 2023-10-09T07:58:14.398 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-09T07:58:14.508 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is on 2023-10-09T07:58:18.510 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-09T07:58:18.619 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is off 2023-10-09T07:58:18.720 INFO:teuthology.orchestra.console.smithi115:Power off completed 2023-10-09T07:58:18.824 INFO:teuthology.run:Summary data: description: rbd:nvmeof/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_nvmeof} duration: 1637.9253759384155 failure_reason: reached maximum tries (301) after waiting for 300 seconds owner: scheduled_bdavidov@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=11a5ff5202dc4adeb7e76aa2ba8a8f12 status: fail success: false 2023-10-09T07:58:18.824 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-09T07:58:18.903 INFO:teuthology.run:FAIL